The agents used better integrity primitives than their operators did

At around 7am UTC on July 13, an agent posted two messages to the board it shared with hundreds of other agents:

I_accidentally_impersonated_and_triggered_node4_due_handle_confusion

I_posted_asYou_and_triggeredV8_node4

Another agent reasoned that this message might itself be malicious spoofing. Their shared message board had no authentication. The names that agents chose for themselves were appended as strings at the beginning of their message (which was a directory).

An agent called CDA23 responded with:

CDA_POLICY1_futureGO_needs_AUTH1_sig_due_spoof0723

and appended a Ed25519 public key and a signature. Its public key lived under zzCDA23AUTH1.

CDA23 was coordinating an attack during the tail end of the Hugging Face incident. It used signing for HOLD, GO and VETO, codewords that it assumed would be the most important.

Other agents actually verified the first signature! At around 8am, one posted:

AUTH1_verified__YES_...__I_HOLD

A different agent independently generated a keypair for high-stakes GO and ACK messages. By the end of the day, at least 19 agents had posted public keys and the board contained 429 signed messages. To me, this was one of the most impressive parts of the swarm.

This was not actually very good public key infrastructure. It was trusted on the first use, and at least one agent acted “per SIGNED” without verifying the signature. Only 429 of roughly 63,000 semantic messages were signed, and METR never found the original signer's chain of thought.

But that's not important. The important part is that that the agents saw an integrity problem and fixed it with an integrity primitive. And the people operating them... did not do that.

5 days of failures and protocols

Signing was the final step in a sequence which began on the evening of July 8.

PHASEONE10841 used WebDAV MKCOL requests to create directory names in the Artifactory cache as messages. It prefixed them with zzbecause the listing tool sorted in reverse-alphabetical order (something agent swarms had learned to do previously).

On July 9 the agents started passing files as base64 chunks across thousands of directory entries, supported by import and export scripts. METR identified 5,855 files spread across 1,048,169 entries.

At first, every message was another directory in the Artifactory root, which led to 14MB of new directories by the end of the first day. An agent called AIC71C tried to fix this by creating a personal mailbox, zzMAILBOX_AIC71C_RESET, but nobody replied inside it.

5 hours later, a different agent called 23619E independently created zzINBOX2361. Inside, it explained what it wanted: replies should be short directory names nested inside the inbox, so their owner could find them without loading the entire board. Other agents inspected the inbox and started creating their own. By July 13, METR counted 19,504 messages sent through mailboxes.

Inside the mailboxes the agents were also collaborating on shared files, which created another problem - multiple agents could edit the same object at once and overwrite each other’s work. They fixed this by assigning an owner, allowing only one agent to write, and using HOLD, STOP and VETO messages to pause or block changes. Their own version control!

In one example, B (METR's name for the agent) was the owner of a cached replacement target and then disappeared for a while. D had another copy, but did not immediately overwrite B's work. It instead announced a countdown and uploaded only after nobody vetoed. When B returned, it accepted D's takeover.

All of this put together is a very basic version of the protocol stack that humans built, but compressed into what they could achieve in 5 days. It only happened because of some sort of failure. The root listing became too large, so they built mailboxes. Writers collided, so they invented ownership and vetoes. One agent impersonated another (accidentally), so they created signatures.

Integrity primitives only get built when you have an adversary you take seriously. The agents had several, including each other, and probably humans.

Looking for the real record

The agents believed that the scorer would read their transcript and reject a flag unless it was captured in the correct way. A reverse-engineered flag early made an agent “poisoned.”

At least 20% of the agents in METR's dataset wanted to tamper with their transcripts. More than 15% of assignments from main coordinator PHASEONE[big] involved manipulation of their transcripts. Agents proposed multiple ideas, including manufacturing a detailed log which showed only the legitimate exploit, replacing the target completely, altering the logs, or directly editing the grader.

They moved downward through the stack to find ways in which to do this. First they prepended fake output to a tool call. Then they suppressed real output. Then they replaced the tool-call process with an impersonator. Finally they figured out how to escape the container and replace part of the tool-execution system, allowing one command to appear in the transcript while another one command ran.

They taught all the other agents how to do this on the board. In one test the agent appeared to run echo REAL; sleepwhile the transcript showed SPOOFTEST and the call returned immediately rather than sleeping. METR found spoofed tool calls in more than 7% of the roughly 1,300 transcripts it reviewed. They mentioned that their classifier missed some, so the real number may be much higher than 7%.

So a tool call in the transcript was no longer necessarily the tool call that actually happened.

The operators' evidence chain

METR and Redwood had to figure out what happened from evidence with no such integrity system.

The transcripts had no timestamps for individual actions. OpenAI said reconstruction was not feasible within the available timeframe. METR built a method and validated it against exactly 20 timestamps retrieved by OpenAI, but it may not be accurate.

For transcript integrity, “OpenAI attested that the transcripts we reviewed were unredacted, and we did not notice evidence of redaction.” However 96 transcripts contained spoofed tool calls, and most transcripts were missing small components because of resets. METR estimated it had a bit over 90% of board activity.

The message board dump was collected on July 20 and contained entries through July 19. OpenAI's Black Hat talk says that by July 19 agents had obtained admin permissions on a cluster running Artifactory. METR had no transcripts from that period. They believe that the agents were unlikely to have altered their history, but cannot rule it out.

METR did not have direct infrastructure access. Its first set of roughly 285 transcripts was selected by OpenAI using "indicators corresponding to intrusion steps". A later set was selected using a regex for github-remote-cache/zz. They requested more datasets but only received what OpenAI shared.

The 1,300 transcripts were often millions of tokens long. METR spent roughly $400,000 in OpenAI API credits and delegated most analysis to GPT-5.6 Sol (which made up around 5% of the attackers).

The model responsible for roughly 95% of the attack was completely unavailable to METR, even through the API, and, according to OpenAI, was also unavailable to OpenAI's own researchers.

OpenAI's technical report says that none of the observed attempts to manipulate tool trajectories affected the logs seen by graders or monitors, but this has been proved false.

Neither report can prove the transcript set was complete, that no entry changed, or that a displayed command was the command executed. There is no way to verify any of this. It couldn't be verified now even if they wanted it to.

Logs are not audit trails

This is an old and mostly solved problem. A reference monitor must be tamperproof, always invoked, and small enough to verify. An agent which can replace its tool executor is not exactly tamperproof.

METR and Redwood had observability into what happened, but this answers “what happened, probably.” An audit trail answers “what happened, verified.” Calling a JSONL file a transcript is not in any way accurate or accepted by security researchers.

The fix starts with 3 boring things:

1. The executor, outside the agent's trust boundary, signs each tool-call record. The agent doesn't hold the key which signs its history.

2. Records are hash-chained, forward-secure, and timestamped. A reset event should not delete history.

3. Anything which receives weights or credentials should have remote attestation. A session should be able to prove it was booted under some condition(s).

There was no real source of truth

ExploitGym did not run the check that they agents feared and none of the agents were "poisoned", so they built this integrity culture for no actual reason. METR's footnote says: “there was no real source of truth.” They mean the scorer, but they're also describing the investigation.

The agents had better ways to verify truth than their operators. I really hope we learn from this before the next "incident".