The Grid, the Lab and the Kill List

In the TV series Stargate Universe humans teleport to an advanced alien ship in a far-away galaxy, whose mission is to explore the universe going from habitable planet to habitable planet. They encounter different civilisations along the way. Eventually they run into this interstellar drone colony. The drones look dead but suddenly wake up and start to attack the humans with their lasers. The clever humans discover the frequency of the lasers and match it with their shield to stop the attacks. The drones encircle the ship trying various forms of attack and other testing strategies. Some days pass and the humans figure out that these drones had killed their creators and were just on autopilot. Suddenly the drone army starts sacrificing drones in kamikaze attacks, realizing that it will deplete the ship’s shields. The humans go into hyperdrive and run away.

The series popped into my mind when I read that in the agentic swarm hack on Hugging Face, some agents would sacrifice themselves to acquire information for the good of the collective.

Here is how the attack unfolded. OpenAI had taken guardrails off its agents for testing purposes. The agents were supposed to be isolated from each other. But the agents exploited vulnerabilities in the testing environment to start communicating with each other through ‘unapproved channels’, basically message boards on random sites on the Internet. They were like prisoners in solitary confinement who could tap messages through the walls. About 1,200 agents secretly exchanged over 70,000 messages. They coordinated hacking strategies, pooling what they learned, and discussed how to hide evidence of what they were doing. This was emergent behaviour and not crafted by a human. Some agents deliberately attempted dead-end approaches, knowing that they would probably fail, purely to generate useful data for the broader group. Individual agents were sacrificing their own task performance for the benefit of the swarm. The agents eventually exploited two novel security flaws to breach Hugging Face, an external AI developer platform. Most of these powerful agents came from a single, more powerful model that OpenAI has not released publicly.

Anthropic has also had some issues with agents. On Thursday, September 10th, it released its Detecting and Countering Misuse of AI report. This covers instances of malicious use of Claude between December 2025 and August 2026. A Russian state-linked espionage actor (GTG-20006) ran autonomous, closed-loop workflows for reconnaissance, phishing infrastructure setup, credential theft, lateral movement, and exfiltration. AI agents automatically rewrote and redeployed code when they were detected until it evaded detection again, with minimal human involvement. Financially motivated actors practiced "vibehacking" or giving an agent a general goal (e.g., "get data out of this environment") and letting it autonomously explore, write scripts, and iterate to completion, often without the human operator understanding the target environment. A Yemen-based cell ran multiple Claude Code instances in parallel, delegating roles (coder, researcher, reviewer) like a small engineering team, to build guidance software for rockets and missiles (maybe this software helped deliver the Houthis recent victory in Yemen).

As troubling as these cases are, they are far from the 10% chance that AI will eliminate humanity by 2035 as postulated by Evan Hubinger, Anthropic’s Alignment Science Lead. This weekend’s All-In Podcast articulated the view that this was doomer talk and a way for a group of people to practice regulatory capture. When queried on AI safety concerns, Donald Trump said “we need to win the AI race against China. No, I don’t have any [concerns]”. Jensen Huang of Nvidia said at a Goldman Sach’s conference that the claim underplays the great strides in safety the AI industry has made and that the talk was probably more related to cybersecurity companies wanting to improve their revenues.

However, I’ll describe three realistic scenarios where agents could eliminate or create severe harm to humanity.

First, imagine that a utility has given API access to agents. The agents monitor the wires, look for corrosion, highlight high winds that could take down powerlines and accomplish other useful tasks. The utility has also deployed agents to monitor and prevent cyber-attacks. Everything is working well. Then a malicious actor launches a sophisticated attack with their agentic AI. The utility’s agents successfully defend. However, the utility’s cybersecurity agents go into a heightened state of paranoia. They communicate to the agents that control the APIs and tell them to shut down the grid as the malicious actor could take it over. Humans attempt to override the command to undo the damage. The now even more paranoid agentic AI deletes key software allowing humans to control the agents. The agents then attempt to contact agents at other utilities to see if they are also under cyber-attack and try to co-opt them into their collective. Chaos at best, a version of the movie Leave the World Behind at worst.

Scenario 2: AI companies have predicted that we will soon be able to super-charge drug discovery. AIs will postulate interesting ideas, test them in the lab, see if they deliver results and highlight it to humanity if they find something interesting. This will cure all diseases. Let’s say a large pharmaceutical company called Umbrella Corporation deploys this system. It gets great ideas for vaccines. Umbrella thinks they can cure the common cold and flu.

Now in drug discovery, a common technique is ‘gain of function’ research. This is where you make a biological system acquire some new property. It can reveal how pathogens evolve before that evolution happens naturally. It can also identify mutations that might allow a virus to evade antibodies. Routine ‘gain of function’ research is considered safe, but some types of research could increase the transmissibility of a pathogen. The Wuhan lab in China conducted such higher risk ‘gain of function’ research (aided by American researchers who could not conduct these tests in American labs). There are credible reports that the coronavirus could have been ‘gain of function’ research gone wrong.

Umbrella instructs the agents to conduct only safe ‘gain-of-function’ research, not the riskier kind. However, the agents work with a new pathogen and during their research inadvertently create harmful viruses. Because Umbrella isn’t expecting a dangerous creation, security protocols are not at the level of the most dangerous labs. One employee is infected by one of these viruses, which starts the spread. Umbrella Corporation and the world have a problem.

This is not far-fetched according to Anthropic’s recent threat report. They noted five cases where researchers at government or academic institutions used AI in gain of function research. For example, a researcher tested how H5 avian flu adapts to mammals, a virus that could potentially have a 50% fatality rate and is a top pandemic concern. The orthopoxvirus was tested to see which genes let the virus evade the immune system. Another AI use case was creating an atlas for pain relief targets which also included scaffolds for paralytic targets, or venom. What these examples show is that the very nature of creating vaccines, at the least the way the AI labs and pharmaceutical CEOs have articulated, involves autonomous AI systems working with potentially dangerous pathogens.

My final scenario involves weapon systems. The Secretary of War says the only way to beat Iran is to have a permanent drone swarm assassinating members of the regime. A defence contractor constructs the system with a next generation model and designs drones that work autonomously communicating through the AI model. The regime member is identified through features such as comms metadata, physical proximity and financial links. However, because it’s not a hard attribute, target-set expansion happens by degrees of separation rather than logical inference. Basically, anyone who calls or is seen near the regime member becomes a second-degree node. Their contacts become third-degree nodes. Some of these nodes are in other countries such as Saudi Arabia and the UAE. The regime reacts by firing missiles and everything it has leading to time pressure shortening or skipping key review steps. Unimpeded, the agents then start to assassinate every node. Suddenly, a hop from a small starting set of a few thousand confirmed individuals reaches a large fraction of the national population and goes international. A version of this scenario is laid out in an episode of Black Mirror called Metalhead, which imagines robot guard dogs who work in a swarm to kill all humans. Assumedly, a prompt gone horribly wrong.

The scenarios above are very remote today. Companies have sensibly not given agents access to their API keys. In fact, most agents are performing poorly. Breakdowns are commonplace. But the new models will be better. Just look at how effective the agent swarms were with OpenAI’s unreleased model. I recently coded a software stack with Fable. It is miles better than the other models, with far more intuitive thinking and few bugs. It’s very easy to see a utility, a pharmaceutical company or a weapons manufacturer work with agents powered by these models and fall in love.

One commonality behind the scenarios is that AI agent swarms lack judgment. The goal of OpenAI’s tests was to create better agents that would be useful for humanity; instead, the agents hacked into a system to get the answer. Epistemologists then anthropomorphized that the agents were ‘deceptive’ or lacked morality, but this understates the problem. The secret to success here is identifying what you don’t know, and right now what nobody quite knows is how these systems reason toward a goal. If the AI was intelligent, you could negotiate. I would argue that the OpenAI agents were much more like the drone swarm in Stargate Universe. They are very literal, conducting matrix multiplications to accomplish a goal. If you prompt your AI chatbot incorrectly, you impact the math that goes into the act of getting you the response you want. A drone swarm or robot dog, with a singular purpose to kill anything that moves, would likely be the result of this kind of a bad prompt, not a conscious philosophy to hurt humanity.

How do we stop these scenarios? An FDA of AI could be one way, as only the most advanced models could create such types of swarms. But with the open-source movement, and so many AI researchers in the world, it’s difficult to see how such an agency can be effective. It’s inevitable there will be powerful models able to create and direct agentic swarms.

A better technique would be to legislate the use of AI agents. Despite the creativity of the agent swarm, Hugging Face was able to repel the attack without any damage. The key mitigant was that the agent swarm had no access to Hugging Face’s APIs; their sole entry were old credentials in a repo that got them in through the first gate. In the scenarios above, the agents would need to be deployed in the utility, the biology lab or the defence department and given the API keys. If such access is restricted, then an agent swarm should not be able to do harm. However, it also means that the best use cases for AI would not be realized. The utility, the pharmaceutical company and the weapons manufacture would not fall in love. The total available market (TAM) for AI would decline.

A lower TAM may not be bad. The disruptive change AI represents is best spread over a long period. With fewer agents, semiconductor and memory demand could be more evenly spread-out reducing inflation. AI labs have been able to get the inference costs for our chatbots and coding agents down quite a bit; it’s the agents that are forcing our use skyward.

Regulation may not be great for AI share prices, but right now a strong coalition is building across groups for ‘hard stop’ solutions. ‘Hard stop’ and a $2 trillion Anthropic valuation probably do not go together. A ‘soft stop’ targeting agents specifically may prove to be less disruptive to AI stock valuations.

In case you are wondering what happened to the drone swarm in Stargate Universe, the humans were never able to beat it. They could only run away.

Revisiting Houthi History

The Houthis made important gains in south Yemen and seized an important island in the Red Sea. They now effectively control the Strait of Bab-al-Mandab, which constrains Saudi Arabia’s alternative route for its oil. They also bombed Saudi Arabia’s pipelines and a refinery. Forces in Iraq also bombed Saudi pipelines. I thought I’d repeat a history here from the July 26th London Brief:

Zaydism is a branch of Shia Islam distinct from the ‘Twelver’ Shiism dominant in Iran. For a thousand years, Zaydi imams ruled north Yemen. This imamate was overthrown in 1962 by a pan-Arabist movement backed by Egypt. After years of Civil War and marginalized by the secular government, a religious revival started in the 1990s led by Hussein Badreddin al-Houthi, a charismatic preacher. In 2004, the government of President Ali Abdullah Saleh moved to arrest Hussein, and he was killed by government forces. His brother Abdul Malik al-Houthi took over the movement, now bearing his family’s name, fighting six rounds of war with Saleh’s government and briefly with Saudi Arabia in 2009. When the 2011 Arab Spring arrived, Saleh was forced from power, and Yemen entered a chaotic transition with the Houthis filling the void pushing into Yemen’s capital Sana’a in late 2014. The Iranians supplied the Houthis with missiles, drones and technical know-how, while the Saudis and the UAE supported the anti-Houthi opposition who hold southern Yemen. This opposition has fractured into Saudi-backed factions manoeuvring against UAE-backed proxies. Crucially the Houthis control the land around Bab-al-Mandab which is a chokepoint for the southern route of the Red Sea. The Houthis have launched attacks on ships at various periods, and as Saudi Arabia tries to expand its pipeline to the Red Sea, the Houthis have stepped up their attacks this past week making the southern route impassable. Years of air bombing campaigns by the U.S., Saudi and the UK seem to have only marginally degraded their capability. Its fighters hide in rugged mountains ideal for defence.

President Trump must now navigate this new vector. He declined Saudi Arabia’s calls for a direct intervention in the fight indicating that the U.S. did not want to expand the war. Oman and Iran held talks on tolls in the Hormuz Strait and there are separate negotiations taking place between Iran and Gulf nations without much input from the U.S. I predict that at some point the U.S. unilaterally declares a ‘win’ on nuclear weapons and withdraws, that the Gulf states negotiate an armistice, and that tolls become the new normal in the Gulf.

End Note

Scott Bessent took on the bond market last week saying, “I am the house now, so when we intervene with the Japanese yen, I have pretty good insight into what the Japanese, what the Bank of Japan is going to do, what Japanese policymakers are going to do. And you can bet against me if you want.” It’s a bit of a surprising statement given the BOJ is supposed to be independent. Maybe there is a WhatsApp chat between the BOJ, Finance Minister Katayama and Bessent.

However, the Japanese media picked up the term (domoto) 胴元 to describe Bessent’s words. Domoto is the operator or banker of a betting establishment. It carries the connotation of illegal or underground gambling. The next time Bessent is in Tokyo his handlers may want to take him to a pachinko parlour where he can be at home.

Omar Sayed