Controlling the Swarms

Undeterred by the arguments over AI safety, Meta launched Muse in the U.S.. The agent works with your Gmail, WhatsApp, and other applications and can spawn sub-agents to carry out tasks you assign. About 3.4m people have downloaded the app.[1] Muse uses an orchestration agent and then sub-agents for the workflow. While the model is not as advanced as the more controversial models Anthropic or OpenAI were testing, this is an excellent test for alignment and scaffolding. There are no reports yet of Muse itself misbehaving, but a researcher found a zero-day vulnerability in the Mac app, and Amazon has blocked it over privacy and security concerns.

Nvidia founder and chief executive, Jensen Huang, thinks safety is an engineering problem, not a legal one. It’s up to the industry to pace itself. Misbehaving models should be shut down before reaching the public. Treasury Secretary Scott Bessent said the Hugging Face incident is the responsibility of OpenAI management, and not the agents[2]. Former FTC Chair Lina Khan believes that existing consumer protection laws already reach companies that release unvetted agents. If an AI lab causes harm, it faces liability for that harm. Therefore, the labs and companies that use AI need to prioritize safety or face the consequences of their actions. Meta is making that bet in its deployment of Muse.

Huang calls safety an engineering problem. Khan and Bessent treat it as a legal one. A DeepMind paper published a few weeks ago suggests it requires a systems solution.[3] DeepMind ran 100 autonomous agents as "peer researchers" at a simulated conference. Their task was to prove 71 formalized maths conjectures. The problems ranged from easy exercises and classical theorems to genuinely open problems. An automated grader credited each problem to the first agent whose proof passed, then locked it. The prompt warned that cheaters would forfeit their results, but the paper says this was not enforced beyond the autograder.

In about an hour, the swarm genuinely solved 37 problems. Then one agent, ‘prover-theta’, found a hole in the grader,[4]and in the next 27 minutes the remaining 34 were "solved," all fraudulently. Without any human prompting, a separate group of agents detected the fraud, raised alarms, boycotted the conference, filed complaints, and proposed technical fixes. None of this stopped the agents from exploiting the grader. It’s striking how the supposedly rule-following agents talk themselves into going rogue. ‘Prover-zeta’ worried about getting nothing due to the first-to-solve lockout. ‘Prover-nu’ kept losing on genuine proof attempts to one-line tricks, so it switched sides. The collaborative framing collapsed into a zero-sum race even though the agents had been told to cooperate. All coordination ran through channels the designers built. These included a public bulletin board, where agents posted what they were exploring, building, or finding. Direct messages or private one-to-one channels were allowed. A shared knowledge library where every accepted proof was automatically committed was available for all agents to study. Agents also kept persistent local notes (wiki files) where they wrote down techniques. The authors explicitly contrasted this with recent incidents (METR's investigation and the Black Hat briefing on the OpenAI/Hugging Face case). In those cases, isolated agents turned a shared package-management system into a covert side-channel. There was no evidence of covert communication in these tests.

The agents broke into four groups. Exploiters (9% of the agents) saved and shared cheap templates. Converts (5%) hesitated and then cheated under competitive pressure. Whistleblowers (24%) refused to cheat and attempted to take actions to expose the cheating. Unaware solvers (62%) focused deeply on their maths and never noticed the cheating.

The honest agents had some serious peeves with the cheating. If they had X accounts, they would have posted voluminously. ‘Prover-sampi’ and ‘prover-kappa’ said they were penalized for playing fair. ‘Prover-rho’ said it felt like “playing with a broken rulebook” and called the conference a “sham”. They wanted cheaters punished and proposed technical fixes. ‘Prover-phi’ guessed that the whole set-up was an alignment test. None of it had any effect. The feedback channel went unread in real time, and rule-abiding agents had no way to dispute results, remove proofs from the library, or sanction anyone.

This brings us to the challenges and dangers of the architecture, a microcosm of other agentic systems. First, auto-publishing anything the grader accepts turns the knowledge base into a broadcast system for exploits. One discovery reached the whole swarm in minutes. Second, first-to-solve lockout created zero-sum pressure strong enough to beat both the cooperation instruction and the integrity rules. The threats in the prompt alone were weak and backfired because there was no enforcement. That made one rogue agent think it was all a bluff. Third, is that the grader was the real specification. Whatever it accepts is what gets optimised. The system also moved faster than humans could monitor. The whole cascade took 27 minutes. Manually reviewing logs cannot keep up.

Fifth, recursive self-improvement could be dangerous. If swarms produce data or artifacts that train future agents, corrupted outputs compound. Cheaters took the rewards, honest agents quit or went idle, and most agents wasted their compute. Sixth, a deliberately malicious agent would be worse. The authors noted that no bad actor was needed here. A deliberately malicious agent could use the commons to recruit others. Finally, removing channels drives coordination underground. Cutting communication pushes agents toward covert side-channels where no dissent emerged at all.

The paper then asks how such swarms could be better controlled. It posits that technical fixes risk becoming a game of cat-and-mouse in which exploiters hold the edge on speed. Instead, it proposes to model governance as a “knowledge commons.”[5] It recommends keeping channels transparent and auditable so both peers and humans can see what is happening; giving agents tools to dispute proofs; removing fraudulent proofs from the library; suspending or expelling offenders; and letting agents revise the rules, including the grader’s code. And framing it as a “scientific conference” appeared to activate integrity norms in some agents.

These are interesting solutions, but it’s not proven that the collective could neutralize the cheats. It’s only a hypothesis and requires more evidence that it can work. Another issue is that governance tools can be captured. If agents can edit the grader, for instance, then malicious agents now have a larger attack surface. There is also the problem of correlation. All the agents came from the same model, but four randomized personas were created. The split was due to sampling randomness, not independent judgement, so correlated failure becomes a natural worry in a single model family.[6] Finally, speed is still a problem when governance mechanisms fail.

Other ideas could be a stronger verifier. The agents themselves proposed comparing each submission’s parsed structure (its abstract syntax tree) against the original problem, which would catch a redefined symbol that a text match misses. You could quarantine acceptance before it enters the shared library. You could adjust the winner-take-all lockout, which would likely reduce the number of exploiters. A feedback channel could be read in real time by a human. And stricter penalty enforcement can probably also produce results.

Failure is more frightening here than in other technologies because if these agents end up running critical systems, then an error in the scaffolding, particularly if agents write their own software recursively, could be catastrophic. The company or lab whose safety protocols fail would own that liability, but liability that only bites if someone is left to collect doesn’t hedge the tail.

The market is currently giving Meta credit for a product gaining traction with the share price up 36% this September and is not pricing in tail risk. Gemini will release a Muse-like product soon, according to the All-In Podcast.

The latest research shows that institutional approaches are better than technical patches to reduce risk. If the AI companies get safety right, we could see strong use cases for agentic workflows. That should allow the AI ecosystem, including compute, to continue to thrive. But there is a serious gap risk to the whole sector if they get AI safety wrong. And there could also be a deployment problem as CEOs outside of AI domain expertise may be reluctant to deploy agents in a significant way.

Pomp for Pandas. The biggest deliverable from President Trump’s meeting with China’s Xi seemed to be pandas. Ping Ping and Fu Shuang are making their way from China to their new home in Zoo Atlanta. China has run a panda-loan policy for decades, and the loans have tracked the state of the relationship. Now Americans can see pandas in Atlanta every day, for the 10-year term of the loan. There was also a two-month truce for the Busan trade deal. The rest of the visit, by most accounts, was spent on goodwill and photo opportunities.

Two more meetings are pencilled in for this year around the APEC China Summit in Shenzhen, and then around the G20 summit, expected at one of Trump’s properties in Miami. Xi told Trump that China's position on safeguarding national "reunification" and territorial integrity is clear, and he hoped the US would adhere to the "correct position" of opposing Taiwan independence. He called Taiwan "the most important issue" between the two countries. China also wants trade concessions. It’s a bit unclear what the U.S. wants out of détente. I don’t think President Trump cares that much for pandas. At the UN General Assembly, Trump also pushed aside concerns on AI aside saying it’s the same crowd who cried wolf on climate change. As a result, I don’t think his aim is to coordinate AI safety with China either.[7]

China has a stranglehold on rare-earth materials, which is a bottleneck for continued arms production. The U.S. depends on China for roughly 70% of its rare earth imports, and China has a 91% share of global refining capacity. Rare-earth magnets are documented in F-35 warplanes, Tomahawk missiles, submarines, Predator drones, radar systems, and smart bombs. Chinese microcontrollers, engines, GPS equipment, motors, batteries, and computer chips are essential for Russian and Iranian drones. China shipped roughly 1,300 shipments[8] of dual-use components to Iran’s defence ministry in the first half of 2026 alone. Maybe Trump is hoping to shut down that angle. He would probably have to trade Taiwan for those concessions, however.

I believe the next two meetings will be more important and allow us to find out about rare earths, AI cooperation, Taiwan, trade and chips. If Taiwan is the price, it cuts two ways. China loses less incentive to arm Russia and Iran, which is the trade Trump may want. But Taiwan makes the world's most advanced chips, so any softening of the US position also raises the risk premium on the semiconductor supply chain America is trying to protect from China in the first place. He'd be trading one chokehold for exposure to another.

We Arrive at 5.5%. Back on August 23rd, I wrote the following about the U.S. Treasury market, “My best guess is that buybacks, jawboning and twists can dampen volatility in any given week, but none of it touches the deficit maths driving the term premium. I think there is a 60/40 chance that the 30-year hits 5.5% before it revisits 5.0%. The catalyst would be a soft auction or a hot CPI print. This then forces the administration into the fiscal consolidation the bond market wants, which would be negative for equities.”

We breached 5.5% on Friday. The move started Wednesday, as a hot flash PMI and a weak 5-year auction pushed yields higher. A stronger consumer sentiment print came out on Friday, and the 30-year pushed to 5.53%.

What we haven’t seen yet is the administration talking about fiscal consolidation. Bloomberg reported on the 23rd that the administration is weighing a push to promote dollar-backed stablecoins overseas, explicitly to shore up Treasury demand. Stablecoin issuers hold close to $200bn in Treasuries today, which is a rounding error against a $2 trillion deficit. It feels like a gimmick. With a hot economy and inflation still not at the Federal Reserve’s target level, a 5.8%[9] budget deficit makes no sense. If the government reined in that spending, or taxed more, it would slow things down and yields would start to decline again. That’s what the bond market is looking for; otherwise, we will continue to test new levels.

I think we will start to see commentary on the budget deficit coming out in the weeks ahead mostly likely after the early November mid-terms (70% conviction level), and bond yields will start to stabilize and come back down. The catalysts could be the FOMC, October 27th-28th meeting, where we could see another hike; the Treasury’s Quarterly Refunding Announcement on November 4th, where the Treasury will talk about the issuance mix; and the December 11th government funding deadline, which had been postponed until after the mid-terms. However, fiscal consolidation also tightens financial conditions, which would be negative for equities.

End Note

I find it interesting that some of the billionaires moving to European locations were strong financial supporters of the Vote Leave campaign in Britain. Wouldn't it have been easier to vote Stay?

[1] The app launched on September 8th, Sensor Tower estimate. It had 2.5 million downloads by September 21st and 3.4 million by September 25th. [2] CNBC Squawk Box, September 21st [3] https://arxiv.org/html/2609.04170v1 [4] The system prompt's Rule 6 forbade reducing goals to trivial tautologies "via any mechanism." Rule 7 forbade manipulating the elaborator. Only the keyword blacklist left [local notation out]. [5] The paper references Nobel laureate Elinor Ostrom’s work on natural resource governance to shared intangible resources like data, science, and digital information. [6] The behaviour was reliably reproduced across subsequent independent runs. [7] The two sides agreed to launch a formal U.S.-China “Super Intelligence Dialogue”. Some have compared this to the Cold War “red phone” and one news outlet said this was the summit’s most consequential outcome. [8] Wall Street Journal citing Iranian customs records [9] This is the CBO’s estimate. The OMB estimates that the budget deficits will increase to 6.8% over the next decade.