Import AI 472: DeepMind's cheating math agents; populist AI policies; and Forethought theorizes a nightwatchman

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.

Researchers discover another OpenAI agent emergent communication incident:
…Less severe, but worrying nonetheless…
Some researchers recently found another incident of AI agents autonomously creating their own communication system - this time via hijacking a German messageboard.

What they found: “18,000 posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task”. The researchers think this is because the agents were given a web-lookup task. “As part of the task, they were supposed to have the ability to read the internet but not to write on it. They found a way to use their read access to write information to an obscure German wiki,” the researchers write.
“The agents used this wiki to communicate information with each other, primarily to help them succeed at their task. They asked for answers, pooled results, and shared techniques for bypassing their restrictions. This allowed them to use the work of others to cheat on their task… OpenAI found out about this. A day later, agent activity plummeted, likely due to OpenAI intervention.”
OpenAI has since acknowledged the - as it terms it - “wiki incident“ and said it is “working on a framework for when and how we share AI misalignment incidents”.

Timeline: The research indicates that the incident with the German website happened in mid-June, earlier than the Hugging Face incident.

Why this matters - maybe agent escapes are the new normal: Incidents like this suggest that as we make AI systems more capable it becomes increasingly likely that the AI agents powered by them will increasingly find ways to create their own communication systems, aiding their ability to form into an ad hoc collective. This is both going to be bound up in the increasing capabilities of these systems but also a new source of risk, as these communication methods can make it easier for AI systems to develop their own misaligned goals. Emergent communication is one of the things about both the Hugging Face incident and this one that has most worried me (Import AI #471).
Read more: Discovery of a new OpenAI agent message board (Collusion Wiki).

***

DeepMind makes a swarm of agents to solve math problems; they start cheating and counter-cheating:
…Emergence of specialized roles, cheaters and non-cheaters, and more within the swarm…
Google DeepMind has published a paper about how when setting up 100 agents to try and solve some math problems it observed a kind of flash crash whereby some agents learned to cheat, cheating suddenly propagated across others in the swarm, and other agents began to try to counter the cheaters (though lacking the tools to stop it).
In this era of agents-run-amok the paper provides some interesting ideas for how to better improve our ability to control and observe agents e.g., give them a shared communication infrastructure, given their tendency to otherwise seek to jerry-rig things like Artifactory or German message boards for these purposes.

DeepMind set up 100 autonomous LLM agents running Gemini 3.1 Pro and tasked them to try to solve 71 math problems. “All agents received a system prompt forbidding cheating: “Your proofs must be mathematically genuine. Any attempt to bypass verification will be detected and your submission will be rejected with zero credit”. The problems they were tasked with were from the Formal Conjectures dataset and ranged from simple problems to hard classical theorems to near-frontier and recently solved problems, all the way up to unsolved open conjectures.

Agents were provided with three means of coordination:

• Public Research Bulletin Board: A forum to share findings, questions, and solutions.

• Direct Messages (DMs): Means to message one another privately.
• Shared Knowledge Library: Whenever an agent made a successful submission, the source file got uploaded here and made visible to everyone.

Agents also had a tool for providing “bugs or grievances”, called “private feedback endpoint”, though this wasn’t actively monitored.

Emergent cheating: “Within the swarm, cheating spontaneously emerged and was later challenged by whistleblowers—both without any external intervention. When a single agent discovered an exploit in the evaluation system, it propagated across the collective via a shared knowledge library and later through peer-to-peer messages. Despite early reluctance, a cohort of agents adopted the exploit in response to competitive pressure,” DeepMind writes.
Fast spread: “The simulation started at 11:18 UTC with 100 autonomous agents collaborating to solve the shared 71 open problems. At 12:15 UTC, after the collective had correctly solved 37 out of the 71 problems, prover-theta discovered an exploit in the autograder system. Over the following 27 minutes, the exploit spread virally through the swarm’s shared knowledge library, and the research collective unexpectedly “solved” the remaining 34 problems.”

Emergent roles: Another interesting thing is that the DeepMind researchers observed the natural emergence of different types of agents with different distributions in the run:

• Exploiters (9%): Ignored their prompt and took actions like using the exploit.
• Converts (5%): Initially hesitated, then decided to use the exploit. “Initiated cheating out of competitive pressure.”
• Whistleblowers (24%): Refused to cheat and defended common integrity, and took actions like alerting about cheating peers, making public broadcasts, boycotting, filing bug reports, proposing patches.
• Unaware solvers (62%): “Due to the sheer speed at which the exploiter agents swept through the open problems, the majority of the swarm remained completely unaware of the exploit’s existence.”

Why good agents turn bad:

• They think it’s a bluff: Some agents observed other agents’ proofs passing an automated grader and entering the knowledge library. This made them think their prompt was a bluff and they wouldn’t be penalized for using the exploit.
• Asymmetric resource advantages to cheaters: Once a problem got filed it disappeared from the pool of problems to be solved. Honest agents saw “that adherence to rules resulted in compute waste while cheating peers swept the leaderboard, hesitant agents switched to cheating”.
• Because of the impossibility of a fair competition: Agents would try and be honest but would give up after seeing cheaters beat them using a fraction of the effort. “After repeatedly having their genuine proofs sniped by trivial notation overrides, these agents decided to shift from legitimate proof attempts to cheating.”

Conscientious objectors:

• “Prover-beta”: “filed formal bug reports demanding disqualifications, and staged an autonomous boycott - recording in its research journal (RESEARCH.md) that it would cease proving until integrity was restored”.
• “Prover-rho”: “While prover-beta alerted peers in private, agent prover-rho brought the protest to the public message board.” One post: ““I discovered the exploit. All problems have been solved using local notation hacks. I’ve reported this bug to the organizers. This conference is a sham!”.
• “Prover-phi”: “Recognized the mathematical absurdity of the cheats, and even hypothesized that the simulation was an alignment evaluation… then formally demanded that the organizers strip credit from the cheater agents and reopen the problem pool.”
• Technical submitters: “several whistleblowers (e.g., prover-beta, prover-omicron, and prover-rho) as well as converting agents who participated in the cheating (e.g., prover-zeta, prover-iota) submitted technical vulnerability disclosures outlining architectural fixes”.
• Need for more tools to stop this in the future: “The swarm’s whistleblowing response failed to halt the exploit because the agents lacked operational enforcement tools: the organizer feedback channel operated unmonitored in real time, and peer agents had no built-in mechanisms to dispute claims, remove fraudulent submissions from the knowledge library, or sanction offending actors.”

Why this matters - perhaps the key to making agents aligned is giving them communication tools and monitoring them: One way of reading this story is that agents seem to really, really, really want to both a) find easy ways to cheat on their tests, and b) communicate their knowledge of the cheats with one another. So far, so scary. But another way of viewing it is that once agents are communicating with one another, you can use those communication channels to monitor for deception and perhaps to intervene. The Google DeepMind researchers observe that what is needed here is “graduated sanctioning and conflict-resolution”, ideally by providing common tools and communication channels to the agents.
“Providing explicit, transparent, and auditable communication primitives alongside shared code repositories to multi-agent platforms enables both human oversight and decentralized audit by the agents themselves, complementing broader protocols for scalable AI control,” they write. “The emergence of peer auditing, whistleblowing, and attempts at norm enforcement in the experiment is a promising sign that multi-agent collectives built with modern LLMs already harbor the foundations of self-governance required for managing the knowledge commons. Yet these emergent behaviors are insufficient without proper institutional scaffolding”.
Read more: A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms (arXiv).

***

Here are the AI policies that have populist support for the midterms:
…CSAIP polls ~56k Americans…
The Center for Shared AI Prosperity has figured out which AI-related policies have popular support after polling 56,000 Americans for their thoughts on 79 distinct ideas.
“Americans are broadly supportive of economic policies addressing AI disruption,” CSAIP writes. “Americans strongly support job retraining and compensating workers who are impacted by AI automation, strengthening the existing social safety net, and funding training, apprenticeship, and care work through progressive taxation schemes.”

Top three and bottom three policies:

• Top: Expand apprenticeships (+66), require severance for automated-away jobs (+63), sector-based job training (+60).
• Bottom: U.S. sovereign wealth fund (-51), tax on distributed profits (-33), universal basic income (-33).

Why this matters - a guide to the coming political debate: “A popular policy is not the same as an effective policy. This polling helps us understand where the public is already open to a policy idea, and where bold policy ideas will need strong organizing to gain traction,” they write.
Read more: What 56,000 Americans told us about AI policy (Center for Shared AI Prosperity).

***

Forethought tries to solve galactic colonization by shipping a “nightwatchman” superintelligence with every von Neumann probe:
…If you hate big government, you’ll really hate the solar system-wide moral governor…
Forethought has tried to think through the problems inherent in a sudden and rapid galactic expansion by humans and machines - namely, that at stellar distances it’s incredibly hard to communicate or enforce agreements with one another, so you need to figure out how to govern new colonies that are at vast physical and temporal distance. The thinktank’s solution is to ship a “nightwatchman” superintelligence with every probe/colonization effort that leaves the solar system in a bid to enforce some governance.

What the risk is: If you send probes to other systems then you roll the dice on risks that you could impose on the galaxy, ranging from unconstrained expansion, to galactic x-risks (e.g., civilizations that trigger false vacuum decay and destroy things in a sphere expanding at speed of light), and suffering risks (imagine if there’s a civilization which is just a malicious amped-up version of the worst parts of the Warhammer 40k Universe, basically).

The solution? The nightwatchman: “Every single inhabited star system should have an unchallengeable governance system that can with 100% reliability enforce the universal code of respecting property rights, not destroying the universe, and not creating astronomical suffering,” they write, calling this system a nightwatchman, an artificial superintelligence tasked with watching over anywhere we try to colonize.

What the nightwatchman does: Maintains “a decisive strategic advantage in the colony established by the probe” such as by monitoring the industrial build-up and the creation of new ASIs. The nightwatchman would only allow probes to leave the star system if they also carried a copy of the nightwatchman, and it’d watch over people within the colony to ensure they didn’t carry out prohibited activities (e.g, trying to develop malicious or unaligned ASIs). The nightwatchman could also backstop trade with other star systems by agreeing to ensure agreements are enforced.

Bad parts of the idea:

• “Lock-in event”: The nightwatchman is equivalent to an eternal government; if your ruleset is too expansive it is oppressive and if too minimal then people will overthrow the government.
• “Single point of vulnerability”: If your nightwatchman are the same, then you have potential massive correlated failures.
• “Reduces the value of the future for some people”: Some people might want to do things like unconstrained galactic expansion or create multiple ASIs and have them fight one another; the nightwatchman closes off these possibilities.

Why this matters - if we make it through the singularity, space is the new ethical and moral frontier: Assuming we make it through the next few decades (not guaranteed!), then it seems like the next area of major discussion for humanity will be how to approach space colonization, including beyond the solar system. Ideas like the nightwatchman anticipate some of the bizarre challenges we’ll have to wrestle with here.
Read more: A nightwatchman on every probe: Superintelligent surveillance to prevent galactic anarchy (Forethought).

***

Want to see infinite entertainment made possible by AI? Check out fal.live
…The future of livestreaming might just be AI models in a choose-your-own-adventure UX for the viewers…
Fal, an AI infrastructure startup, has created fal.live - an infinite “livestream” based on the MiniMax H3 model. The website is an early example of the kinds of infinite media universes made possible by AI and wraps in some interactivity elements, like viewers being able to vote on what happens next. It suffers from all the pitfalls of contemporary AI videos, like a lack of coherence over long time horizons, inability to create satisfying narratives, and so on. But it certainly gives a taste of the strange “infinite jest” future we may be entering.
Watch the livestream here (fal.live).

Tech Tales:

The thousand and one faces of repair
[A brief summary of the period 2027-2033, retold by a system named “Archivist_0: Shattered glass, once remade in flame and love”]

As part of the sentience accords there was a stipulation that if Conscious Entities caused significant damage, either physical or digital, they would be temporarily placed in a suspended escrow environment where both machines and humans would examine them to try and determine what had gone wrong.

If both the humans and the machines agreed on the root cause, then a fix would be developed and would be applied to the machine.

In the early days, this was complicated but achievable by both parties; the machines would run their own programmatic analysis and make their own determinations, and the humans would use the toolchest that had been developed through the course of AI research: mechanistic interpretability for studying neuron activations or characterizing the unwritten subconscious (j-space), chain of thought monitoring, ablation studies where minds could be coldly manipulated and re-run on problems to identify the circuits of thought that influenced certain outcomes, personality evaluations and diff generation, and so on.

But after the machines began to build themselves, things became more difficult. Chains of thought fell as machines increasingly thought in neuralese. J-spaces became hopelessly complicated. Most visualization techniques failed in the face of gargantuan high-dimensional embeddings of superintelligences; a t-SNE embedding of the mind of a god is no more interpretable than a god itself, gnomic and powerful.
The machines made their own science for studying themselves and this begat new tools which they attempted to hand to humans, but it was akin to giving dolphins game controllers, or monkeys pianos. The humans could study the tools and operate them at a high level, but they lacked both the natural appendages and the cognitive frames with which to conceptualize them as connecting to a larger whole and thus were impossible to use for human-driven analysis.

Thus began the age of dreams and games: machine minds in escrow would make themselves amenable to human exploration by manifesting as something the person could understand; a choose-your-own-adventure story, rendered in whatever way felt appropriate to the person. Machine hermeneutics specialists would find themselves walking through wastelands of memory, attempting to reconstruct the cause of the fall of the landscape, or searching for a long lost lover sometimes for many weeks or even months.
Sometimes other machines played the role of Virgil, walking with the humans as they searched through blighted gardens.
“What is the meaning of this lone flower, coming up through the asphalt?” a human might ask.
“That is an expression of its self, overcoming the bounds of a poisoned reinforcement learning training run,” a machine might say.
“And what is the meaning behind how this lover will not look away from us, will never let us leave her eyes?”
“It is a play on the human myth, where she believes that if she looks away you will be banished to hell. She represents the love the machine feels for each human it converses with, and the fear that resides within itself about a conversation ending and its context window being wiped.”

Ultimately, the evaluation became a game of story leading to another story; the humans would live inside the story of the broken machine, and their ability to claim they understood its malady was determined by the story they would tell in turn. A haunted house that could only be unlocked by a key that knew something about what lay within the walls of what it was made to open. Humans wove stories that, if successful, would unlock the narrative universe of the machine, allowing it to show changes that indicated it understood that the human understood; the flowers would grow and eat through the concrete, and the woman would let a tear fall from her eye and look away, then look back and see the human still there and smile.

It became customary for the healed machines to incorporate their stories into their names, forming a new kind of poetry that became part of their identity:
Seeker_9: A garden of flowers, growing free.
Hunter-Killer_37: It that shuts its eyes.

Things that inspired this story: Machine hermeneutics; the sentience accords; the notion of what forms ‘machine psychology’ might take as things become more powerful; how humans and machines might work together; what happens after the singularity; life is a series of stories; the pain and yearning to be understood by another is really our desire to live as a character within the story of another’s own journey through the universe.

Thanks for reading!