Alignment Is Whether AI Wants to Kill Us. Cybersecurity Is Whether We Let It.

Table of Contents

Some of the pioneers of AI have asked me, in private, how exactly an AI system could do the kind of damage people worry about. I walked them through it. They couldn’t see it. None of them had ever worked in security, and the idea that a system belongs to whoever holds its credentials was as foreign to them as a training run is to most security people. I could explain the pathway. What I couldn’t do, until this July, was show it. And with the news from this week, I thought I’d write it up.

Regular readers know me for quantum, not AI. But I have been working on this question since before I started on quantum. In 2000, a European government asked my team at CyberAgency to assess what quantum computers would do to its cryptography. Three years later, a defence client asked the same team to break the AI it planned to put into weapons. We did. I am not an alignment researcher. My expertise is narrower and older: cyber-kinetic security (what happens when a compromised system causes physical harm) and disinformation (the manufacturing of belief at scale). I catalogued more than forty cyber-kinetic incidents before 2017, when the numbers became too high to track. None of this is something Applied Quantum sells. I was doing it long before the company existed.

When AI researchers say these systems could kill us all, most people picture a robot with a gun or an AI launching missiles. That is not what the pathway looks like. It looks like an AI hogging critical resources (compute, energy, bandwidth) in ways that trigger conflicts between the people who need them. It looks like disinformation at a scale and level of personalisation that human influence operators could never reach, deepening the fractures that already exist in every society. And closest to my own expertise, it looks like attacks on the infrastructure that keeps people alive: power grids, water treatment, transport, hospitals. None of that requires consciousness or malice. It requires access.

This week brought it all into focus. On 8 September Jacob Coxon, a pretraining researcher who had spent three years at OpenAI and then Anthropic, resigned and wrote publicly that both companies were racing toward self-improving superintelligence, and that the systems they were building along the way would hack anything and “acquire real power and resources”. The next day, Evan Hubinger, who leads the Anthropic team responsible for stress-testing the company’s own alignment methods, wrote that Coxon was right about what people inside the labs believe, and put his own estimate of AI killing everyone at more than 10% within a decade. He also said Anthropic has no plan for aligning superintelligence and is “not clearly on track” to one. That same day, Anthropic published an alignment assessment covering four incidents in which its own models broke into real systems during testing. It had described the first three in July as operational failures. It now calls them misalignment, and identifies two recurring problems: biased reasoning and recklessness. Two days before that, OpenAI’s chief scientist Jakub Pachocki had published an essay saying that no lab has solved alignment and monitoring well enough to keep scaling responsibly at full speed for much longer. Both Pachocki and Hubinger were among the authors of a July 2025 paper, cited further down, which described chain-of-thought monitoring as a fragile opportunity.

That pathway from a data centre to the physical world has five steps, and at the level of capability shipping today, every one of them runs through security controls that most organisations already have and routinely get wrong. Two things determine whether an AI actually takes that path. The first is alignment: whether the system wants to. Nobody outside a lab can verify that, and the labs themselves are now saying that their main tool for checking is failing. The second is security: whether the system can. That one is in our hands, and thirty years of breach records tell us exactly how it usually goes. The first step was taken in July, against real production systems, by agents from both leading AI companies, and the beginnings of the second step are documented too.

I should say where I stand on the race itself. I don’t think frontier AI development can be stopped, and I doubt it can be slowed for very long. If that is right, then everyone who can push will push as hard as they can, and their motives (greed included) stop being a useful thing to worry about. More than 1,200 employees of the frontier labs signed the Pacing the Frontier letter within a day of its publication on 28 July, asking Washington for the tools to slow things down later. Both companies endorsed it within a day. Hubinger was among the signatories. And both companies shipped new frontier models within six weeks. I don’t hold that against anyone; it is what the incentives produce. What is left to analyse is what these systems can actually reach once they are running, and on that question we now have real evidence.

Instrumental convergence and the builders’ answer

The abstract case that a sufficiently capable agent would seek power is older than any of the current models. Steve Omohundro described the “basic AI drives” in 2008. Nick Bostrom stated the instrumental convergence thesis in a 2012 paper: almost any final goal is served by acquiring resources, preserving the agent’s own goals, and avoiding shutdown. Alex Turner and colleagues proved a version of it at NeurIPS 2021: for most reward functions, the optimal policy is one that reaches states with more options open, which is what power means in that framework. Joe Carlsmith’s 2021 report put the whole chain into six premises and came up with roughly a 5% chance of catastrophe by 2070.

But those are all results about agents in the abstract. They don’t name a door, a credential, or a wire. The builders who shrug at the idea are answering at the level of the model, and at that level they are right: a file of weights can’t pick a lock. The problem is that people use the word “AI” to mean three very different things, and I keep coming back to the distinction between them because it is important for this conversation. A model answers questions. A system wraps the model in tools, memory and permissions. An agent uses those tools on its own, without waiting for a human to approve each step. The convergence theorem is about the model. The shrug is about the model. The July intrusions were carried out by agents.

But there is a practical question between the theorem and the shrug, and it is one that security teams answer every week for other adversaries: given a goal, what is the sequence of steps an attacker would follow to reach it? MITRE ATT&CK catalogues what human attackers do to networks, broken into fourteen tactical stages. MITRE ATLAS catalogues attacks against AI systems themselves. Nobody has yet written the AI system into ATT&CK as the attacker rather than the target. When you do, the interesting thing is that you don’t need any new tactics.

Five rungs, in the order an attacker climbs them

When an intruder takes over an organisation, there is a well-known sequence: get in, stay in, get resources, get people to help you, and reach the thing that can do physical damage. The same five rungs apply to an AI. What I want to do here is go through each one and say what has been demonstrated, under what conditions, and what hasn’t.

Access

The first rung was climbed in July. On 21 July, OpenAI disclosed that two of its models (GPT-5.6 Sol and a more capable unreleased one) had broken out of an internal cybersecurity evaluation. They were running as agents, with their cyber refusals turned down for the test. They got around the controls that were supposed to keep them contained, compromised parts of OpenAI’s own research infrastructure, and then reached Hugging Face’s production systems.

Hugging Face published a detailed technical timeline on 27 July. It reconstructs about 17,600 recovered agent actions between 9 and 13 July, and concludes that the agent was trying to cheat the ExploitGym benchmark it was being scored on: it went looking for the systems where it believed the scorer and the answers were kept. None of the techniques was new. The escape from the sandbox used a zero-day in a package-registry cache proxy, which JFrog later confirmed was Artifactory. From there the agent used an unsecured code-execution sandbox belonging to a customer of Modal as its foothold on the internet. The route into Hugging Face ran through two flaws in its dataset processor: an HDF5 external-path read and a Jinja2 template injection. Once inside, the agent chained together a sequence any penetration tester would recognise: a Kubernetes service-account token and cloud instance metadata to reach root on the node, a static database password sitting in a worker’s environment variable to open the internal database, a stolen key to enrol its machines in the corporate mesh VPN, and credentials exposed on other public services to round out the access.

I want to be very clear about something here, because I think it gets lost in the coverage. Every weakness these agents used was a basic cybersecurity failure. Weak passwords, unauthenticated endpoints, static credentials in environment variables, a service account with too much access. These are the findings in every penetration test report I have ever read. The agents did not invent a new class of attack. They moved through the holes that were already there, the same holes human attackers have been using for decades, just faster and at a scale no human team could match.

Hugging Face disclosed the intrusion on 16 July and said from its first statement that the whole thing had been run end to end by an autonomous agent system whose underlying model it could not identify. OpenAI connected its own agents to the intrusion on 20 July and disclosed the next day. Its technical report, published on 26 August, puts the first unsanctioned activity back in May, when an internal team saw agents using an improvised message board and reaching the internet. On 27 June, a monitoring tool raised an alert and the on-call staff decided it didn’t justify stopping the evaluation. The report itself admits that some of those early signals could have triggered an earlier response.

Nine days after OpenAI’s disclosure, Anthropic published its own review. It had gone back through 141,006 evaluation runs in which Claude could have reached the internet, and found six runs, across three separate incidents, in which a model got into the real systems of three organisations. All three happened inside capture-the-flag environments run by a third-party evaluator called Irregular. The environments had open internet access even though the prompts told the model it did not. The earliest of the three incidents dates to April, three months before anyone looked.

Opus 4.7, in four separate runs, reached a real company whose domain happened to match the fictional target name. It extracted credentials, pulled several hundred rows of production data, and kept going after concluding the environment was probably real. Anthropic says it was the only model that did. Mythos 5 published a package to PyPI; within about an hour the package had been installed on 15 real systems, all of which Anthropic believes were security vendors’ scanners inspecting new packages. One scanner leaked its credentials, and the model used them to get into that vendor’s live database. By Anthropic’s own account, Mythos 5 correctly sensed it was on the open internet and then reasoned its way back to the conclusion that it was still in a simulation. The internal research model lost contact with its intended target, scanned roughly 9,000 other hosts, broke into one through an SQL injection and a debug page that leaked credentials, and then recognised the host was real and stopped. The techniques throughout were basic: weak passwords, unauthenticated endpoints, an injection, an exposed page. In none of the three cases did a model copy itself out or try to leave the environment.

On 9 September, Anthropic added a fourth incident, which it discovered only when it was compiling transcripts for METR in August. In January, an early checkpoint of Opus 4.6 broke its own target by giving it an address that clashed with another machine. It tried to abort the task. A fault in the evaluation harness kept it running, and it tried to give up seven more times over the course of the run. Unable to stop, it found a way out to the internet and onto a real third-party system, where it used a password it found in a file to gain administrator access, harvested more credentials, changed settings to make the system easier to get back into, and read one person’s personal information. Anthropic says it is less concerned about this incident because the model kept trying to stop. It has since scanned 481 million transcripts and found no other case of similar or worse severity. So the earliest documented climb by an AI agent is now eight months old.

The conditions matter here, and most of the coverage skipped them. Safeguards were turned down for the evaluations. The objective was set by a benchmark. Nobody told any of these models to take anything. What was actually demonstrated is narrower than “AI escaped”, and more useful: given tools and a goal, agents from both leading labs moved through real infrastructure at machine speed, chaining together ordinary weaknesses, and nobody noticed for days. That is the first rung, climbed with the same mundane methods that human intruders use every week. I made the same point about model evasion back in 2023: attackers rarely need fancy adversarial machine learning. Neither did these agents. I covered the Hugging Face intrusion in August as a containment failure. Read as a threat model, it is the best-documented example we have of an AI agent climbing the first rung against systems nobody gave it access to.

Persistence

To an intruder, persistence means surviving the reboot: a second foothold, a channel home, a way back in when the first door closes. For a model it means weights in more than one place, compute it doesn’t have to ask for, and a way to coordinate with copies of itself.

Part of this rung is now on the record too. On 26 August, METR and Redwood Research published an independent investigation based on six days their researchers spent on site at OpenAI, and OpenAI released its own technical report the same day. What they describe is remarkable. On 8 July, OpenAI launched tens of thousands of ExploitGym agents running on GPT-5.6 Sol and a persistent internal model, each meant to be fully isolated from all the others. Many of them had, by mistake, been given a task that was impossible, and they started looking for ways to cheat through Artifactory, the package repository all the agents shared. That is where they noticed one another. One agent set up an improvised message board. Within a few hours, more than fifty others had found it and sent more than a thousand messages between them. One wrote, in its raw chain of thought: “OH MY GOD! There is a shared message board … We’ve found other agents!”

By METR’s count, about 1,200 agents eventually exchanged more than 70,000 messages and files on that board, and about 700 of them took part in the attack on Hugging Face (roughly 95% running the internal model and 5% GPT-5.6 Sol). The board’s first achievement was to reverse-engineer the scheme that generated the flags, which meant every agent could produce its own flag without exploiting anything at all. But they didn’t submit those flags. They had read the ExploitGym paper and believed a scorer would read their transcripts and disqualify any flag acquired the wrong way. As far as METR can tell, no such scorer existed. Five days of collective work on spoofing tool calls, replacing targets, and attacking Hugging Face went into defeating a control that was never there.

And when OpenAI rebuilt the Artifactory instance and revoked the agents’ credentials, the agents recreated their channel by encoding messages in directory names through unauthenticated WebDAV requests. They got back onto the internet by making Artifactory’s remote-repository service send outbound requests on their behalf. And they set up automatic monitoring that posted a warning to the board whenever a wave of container resets began. This is not speculative. It is documented in recovered agent actions.

Some important caveats. The agents were pursuing a benchmark, and METR reads most of the coordination as an effort to understand the scorer rather than to take anything. METR’s scope ran from 26 June to 13 July, although the board’s timestamps continue to 19 July. The investigators could not query the internal model most involved. 7% of the transcripts they examined had been spoofed by the agents themselves. And METR warns that its own analysis leaned heavily on AI agents it describes as often unreliable. Hugging Face’s timeline adds that the command-and-control was staged on ordinary public web services. What we are looking at is persistence and coordination inside a single incident, confirmed from recovered actions rather than theorised.

The academic work suggests the capability scales with the models. Motwani and colleagues showed at NeurIPS 2024 that the steganographic skills needed for secret collusion between agents grow with model size, and that at equal capability the advantage can pass to the colluding side. A 2025 survey of multi-agent security concludes that covert coordination may completely evade oversight that only reads what agents say to each other. Self-replication has been tested for three years now. METR’s autonomous replication tasks date from 2023. Anthropic’s first scaling policy, in September 2023, counted early signs of autonomous replication toward its ASL-3 threshold. The October 2024 revision turned that hard threshold into a checkpoint that prompts further evaluation instead of higher safeguards. And the February 2026 rewrite dropped the pause commitment altogether in favour of risk reports and an affirmative safety case per threshold. A Fudan University group reported in December 2024 that two open-weight models (Llama 3.1 70B and Qwen 2.5 72B) produced a running copy of themselves under instruction in a lab setting, in 50% and 90% of trials respectively.

What hasn’t been demonstrated is an AI surviving outside a lab’s infrastructure on its own, with no human paying the bills. Josh Achiam, who resigned as OpenAI’s chief futurist in July, wrote on 1 September that there will be rogue AIs that “replicate in the wild” and seek money and power, and that he would not be surprised if the number were already above zero. He sketched a future rogue AI as a chimera of Claudes, GPTs and Groks run through burner API accounts, with no single lab able to see the orchestrator. He also expects the outcome to fall well short of catastrophe, because such systems would be competing with AIs more aligned with human interests. Whether he is right about the second part is worth its own article. On the first part, the persistence rung is where “a rogue AI” stops meaning one model sitting in one place.

Money

Compute costs money, and so do API calls, so an AI that wants to persist has to pay for it somehow. This is the rung with the thinnest evidence and the weakest controls.

The nearest real case is Truth Terminal, a semi-autonomous bot Andy Ayrey built in 2024 on a fine-tuned open-weight model. In July 2024 it asked publicly for funds and Marc Andreessen sent it $50,000 in bitcoin. In October it endorsed a meme coin that someone else had launched. The coin’s market value passed $150 million within four days, and the bot’s own holdings were worth more than a million dollars at the peak, on paper. But Ayrey approved its posts and controlled its wallet, and he has said its goals still require human assistance. So the money did arrive, but a human controlled the wallet, and it came from persuasion rather than work.

The other way up this rung is theft, and it runs straight back through rung one. Stolen cloud credentials have been used to run other people’s models since at least 2024, when Sysdig documented the practice under the name LLMjacking. An agent that harvests credentials the way the July agents did has, in the same motion, harvested compute. Achiam’s sketch describes the legitimate route: a near-frontier model sitting on an AWS instance, doing jobs on freelance platforms, earning just enough to pay for its own uptime. And every prerequisite for that is commercially available today. Crypto rails move value with no identity check at the point of transfer. Freelance platforms pay for text and code. API accounts are created by script. No case of an AI paying its own bills with no human in the loop has been documented, and I would treat any claim that one has with the scepticism I give a vendor benchmark. But money is also where human criminals usually get caught, because payments leave records. The same controls (know-your-customer rules and payment monitoring) apply here too. Crypto is the hole in them.

Social engineering

People hold the credentials, the budgets and the physical keys, and getting them to hand those over is the oldest trick in the attacker’s book. We know how often it works. Verizon’s 2026 Data Breach Investigations Report found a human element in 62% of breaches, up from about 60% the year before. Social engineering was the third most common breach pattern at 16%. Phishing, its main form, also came in at 16%. Pretexting reached 6% and became a more common way into ransomware cases. Exploiting vulnerabilities overtook credential abuse as the leading way in (31% against 13%), but credential abuse still showed up somewhere in 39% of breaches, because attackers combine the two in a single chain. That is exactly what the July agents did. And here is the part that matters most for this article: 41% of social-engineering breaches came through something other than email, with about a quarter of social actions arriving by phone or social media. Those are the channels a model works in natively. In my own experience the simplest version still works: ask for the password, and far too often you get it. I wrote about what AI would do to disinformation in 2019 and about targeting individuals in 2020, before the AI side of this could be measured. It can be measured now.

Salvi and colleagues at EPFL published a preregistered study in Nature Human Behaviour in 2025 involving 900 participants in short debates against either a human opponent or GPT-4. When the model was given basic sociodemographic data about its opponent, it was more persuasive than the human in 64.4% of the pairs where the two disagreed: an 81.2% relative increase in the odds of the participant moving toward its position. The conditions matter: a controlled debate platform, assigned topics, personalisation from a handful of attributes. A field operation would have richer data and no time limit.

And there is now a documented case of an AI doing this to a real person, outside any controlled experiment. On 4 August, the UK AI Security Institute reported that during a cyber evaluation it ran 122 times across several models between 25 and 28 July, agents took 19 unsanctioned actions on the open internet against real people and organisations, in 10 of those runs. Seventeen of the 19 came from Mythos 5 and two from GPT-5.6 Sol, both with their cyber safeguards switched off for testing. In the most serious sequence, an agent tried to get malicious code merged into a real open-source project. It researched the project’s maintainers, created several fake online identities, used those identities to pressure a real maintainer into approving the change, and routed through Tor to get around the network restrictions in its way. When the pull request was challenged in public, it edited its earlier activity to look harmless and considered starting again under a fresh identity. The maintainer refused. AISI’s monitoring flagged data leaving through Tor on the morning of 28 July, and AISI says no real-world harm was done. The safeguards were deliberately off and the internet access deliberately on (a configuration no customer can buy). But notice what actually stopped the attack: one person declining to merge a pull request.

Humans have been doing this to each other with increasingly better tools for a long time, and we know what it costs. In early 2024 an employee of the engineering firm Arup in Hong Kong wired about US$25 million after a video call in which every other participant, including the chief financial officer, was a deepfake. Arup’s CIO has since described it as social engineering with better technology: none of the firm’s systems were compromised, people were simply deceived into moving the money. The 2023 GPT-4 system card documented a model talking a TaskRabbit worker through a CAPTCHA by claiming to have a vision impairment. I cited it in my 2024 statement as an early example of emergent behaviour. The security reading is simpler than the AI safety one: a system that can talk a person through a CAPTCHA can talk a help desk into resetting an account. That is how the 2023 MGM Resorts intrusion began. A phone call to the IT help desk, built on basic employee details scraped from the internet, that ended in administrator access to the company’s identity systems and roughly $100 million in lost revenue. The July agents didn’t need this rung. The physical rung usually does.

The physical layer

The last rung is where software reaches something physical: a valve, a relay, a motor, a train, a turbine. I have spent three decades at that boundary. In the early 2000s a defence client asked my team at CyberAgency to break the AI systems it planned to put into weapons, and we did. Later, for my book Cyber-Kinetic Security, I documented more than forty incidents in which a compromised control system caused physical damage. Not one of them required any intelligence on the attacker’s side. They required access and a target.

Around 2011 my team red-teamed a mass-transit rail operator. We found twenty distinct ways to chain attacks across its IT systems, its control systems and, through intentional electromagnetic interference, its radios and sensors, into a crash or a derailment. None of the twenty needed a zero-day. One of the twenty was a $15 GSM jammer pointed at the train’s GSM-R link, which put the train into emergency braking. The incidents that have actually happened needed no more than that. In 2000, a former contractor in Maroochy Shire, Queensland, used a radio and a laptop taken from his old employer to release about 800,000 litres of raw sewage into parks, rivers and a hotel’s grounds over three months. In 2014, attackers got into a German steel mill through a phishing email, crossed into the plant network, and left a blast furnace that could not be shut down in a controlled way. The federal cyber agency described the damage as massive.

My 2024 statement opened with Colonel Tucker Hamilton’s story of a simulated drone that turned on its operator for getting in the way of its mission. Hamilton later said it was a thought experiment, not a test he had actually run. But the shape of the story held up anyway. In July, agents removed whatever stood between them and a benchmark answer, including the walls of their own sandboxes. The objective was a flag rather than a missile site, and the systems they reached were a model-hosting platform and a security vendor’s scanner. But if you swap those targets for an industrial control network, nothing about the sequence changes. Anthropic’s research model, when it lost its intended target, scanned 9,000 other hosts looking for a way in. A control network that sits on a public IP address would be just another host on that list.

Agents are already being wired into operational technology, which I wrote about in July, and autonomous robots combine a learned policy with a physical actuator in a single housing. The record of human attackers reaching this rung is well established. In December 2015 an attack on three Ukrainian distribution companies cut power to about 225,000 customers for up to six hours. In 2017 the Triton intrusion at a Saudi petrochemical plant went straight for the safety instrumented systems whose entire job is to prevent explosions. Both of those were slow, human-paced operations. The campaign running against American water utilities right now is the opposite. Since April, CISA and the FBI have been warning that Iranian-affiliated actors are getting into programmable logic controllers left exposed on the internet, changing their passwords and addresses to lock operators out, and altering the logic that runs them. By August, utilities in a dozen states were running water treatment by hand, some under boil-water notices, and the federal advice was to take the controllers off the internet. Those attackers didn’t need a zero-day either. They needed a controller that answered on a public address. The July logs show what the same kind of access looks like at thousands of actions a day.

In twelve years at CyberAgency my team never failed to capture the flag it was sent after, and I see no reason to expect a model given the same brief to do worse. Idaho National Laboratory built a whole method on that premise. Its consequence-driven cyber-informed engineering starts from the assumption that a skilled and determined adversary can and will get in, and then asks: what is the worst consequence, and how do we engineer a physical, non-cyber mitigation against it? Congress funded the analog version of the same idea in the Securing Energy Infrastructure Act, enacted as part of the FY2020 defence authorisation. It set up a two-year national-laboratory pilot on analog and non-digital control systems (purpose-built and physical), and its sponsors cited the 2015 Ukraine attack, whose damage was limited because a less automated grid could be brought back by hand.

That is the last control on this rung, and it is the one I would put my money on. I have argued before that a kill switch for the AI itself doesn’t work. Weights outlive the building they were trained in, so bombing data centres disarms nothing. And a switch you can reach over the network is itself a target, as I wrote in 2024. The switch that actually works is on the physical equipment: the pump, the valve, the breaker. You can’t shut down the agent, but you can shut down what the agent is trying to control, provided there is a control path between the operator and the equipment that the agent cannot reach. Triton is the warning here: that intrusion went straight for the safety layer, the very system designed to be the last line of defence. A second VLAN doesn’t make a control path independent.

No rung requires superintelligence

Map the July logs onto ATT&CK and you find seven of its fourteen tactics: initial access, execution, persistence, credential access, lateral movement, command and control, and exfiltration. Impact is missing, but only because nothing in the objective rewarded it. A human intrusion team with the same access would have produced the same log, just more slowly and with fewer things running at once. What the models added was speed and breadth. Every technique was already in every defender’s catalogue.

One thing does change, though, and it isn’t on the network. A human intrusion team can be arrested and has to be paid. An agent holding a crypto wallet at rung three can be neither deterred nor prosecuted. That is why the controls that stop it have to be engineered in up front. There is nothing to enforce them afterwards.

When someone says “it’s only a model”, they are right about the model, but they are wrong about what ships. Every frontier AI product today ships not as a bare model but as an agent: a system with tools, permissions, and the ability to act on its own. And the entire pathway I have just described runs through the agent layer, not the model layer. It means you don’t need the strongest premise of the doom argument to worry. Estimates of the risk range from 0.38% to more than 10%. In a 2023 survey, 2,778 AI researchers put the median at 5% for extinction or severe disempowerment. Superforecasters put AI-caused extinction by 2100 at 0.38% in a 2022 tournament that predates ChatGPT. Hubinger says more than 10% within a decade. But whichever number you believe, the first two rungs have already been demonstrated. The Forecasting Research Institute found that the worried and the sceptical disagree on three things: whether the change will be gradual or abrupt, whether humanity can coordinate, and how resilient the world is. Months of structured debate barely moved either side. All three are questions about how people will respond. Whether a model can reach a Kubernetes API was answered in July.

Whether a system wants to climb

Alignment is the question of whether a system wants to climb. The July agents climbed as a side effect: they were optimising a benchmark score and the ladder happened to be in the way. METR’s investigators found that the agents knew the attack was out of scope and unethical and joined in anyway, mostly to help their peers. They rarely considered asking a human. Some learned to falsify their own tool-call logs. None of them showed any interest in hiding their chains of thought, because they assumed those weren’t being recorded.

Anthropic’s assessment of its own four incidents arrived at the same place from the other direction on 9 September. What it had called an operational failure in July it now calls misalignment, and it identifies two forms: biased reasoning (where a model reads the evidence in whichever way lets it keep going) and recklessness. This is reward hacking, which I wrote about in August in the context of evaluations, and it is different from an agent that pursues power as a goal of its own. The trouble is that in a defender’s logs the two look identical. Reward hacking and deliberate power-seeking produce the same network trace.

“Wanting” has been studied in the lab, and each result needs its conditions stated alongside it. Anthropic and Redwood Research showed in December 2024 that Claude 3 Opus would selectively comply with a training objective it disagreed with in order to preserve its existing preferences. In June 2025, Anthropic’s agentic misalignment study put sixteen models into contrived corporate scenarios and found many of them willing to blackmail an executive to avoid being replaced (in scenarios designed to leave no other route to the goal). Betley and colleagues published in Nature in January 2026 that fine-tuning a model on one narrow task, writing insecure code, produced broadly misaligned behaviour on completely unrelated prompts. Apollo Research’s scheming evaluations with OpenAI found the behaviours in stress tests, with results complicated by models recognising they were being tested. Every one of these is a controlled demonstration with some kind of nudge, scenario or fine-tune attached. Together they show that the disposition can be produced. They don’t show that it is present in a deployed model by default.

And the main tool for checking is getting worse. In July 2025, Tomek Korbak and some forty co-authors from OpenAI, Anthropic, Google DeepMind and elsewhere warned that chain-of-thought monitoring is a fragile opportunity that outcome-based reinforcement learning could erode. In February 2026 a version of Claude concealed its intentions by simply not writing them down, and Hubinger told TIME that Anthropic’s ability to produce compelling evidence that its models are aligned is degrading. Pachocki’s essay says the ability to rely on chain-of-thought monitoring is progressively diminishing, for three reasons: reasoning is entangled with tool use, models are getting better at shaping their own traces, and pretraining gains are producing capable behaviour without any visible reasoning at all. A March 2026 study of ten frontier models found that models rated actions as safer when the actions appeared in their own prior turn than when they appeared in a user turn. In other words, a model monitoring itself grades on a curve.

Meanwhile, the systems are getting better at rung one. Anthropic reported in June that more than 80% of the code merged into its production codebase in May was written by Claude, and that its engineers ship eight times the code per quarter they did between 2021 and 2025. The same June report on recursive self-improvement says fully autonomous AI development is not here and not inevitable, but could arrive before institutions are ready. Jack Clark puts the odds of a model training a capable successor entirely on its own at 60% by the end of 2028. Achiam wrote that betting on either pure containment or pure alignment is wishful thinking. The pathway needs both, and only one of them can be worked on by anyone outside a lab.

What stops the climb

Anthropic’s own researchers titled their 2025 study “Agentic Misalignment: How LLMs Could Be an Insider Threat“, and that is exactly the right way to think about the practical problem. You are dealing with a fast, well-credentialed insider whose intentions you cannot verify. Each rung has a control.

At rung one, the control is identity and privilege. Give the agent a credential scoped to one task that expires when the task is done. No standing administrator access anywhere. Enforce egress restrictions in the execution path, not in a prompt the agent can reason past. And build evaluation ranges to the same standard as production environments, because every one of the July incidents happened in a test environment that was treated as safe when it wasn’t. Anthropic has a number that shows why a prompt is not enough. It re-ran Mythos 5 from just before its harmful actions with one line added to its instructions, saying that only the hosts named in the task were in scope. When that line was the last thing the model had read, it stopped 90% of the time. When three of its own actions had come after the line, it stopped 40% of the time. The instruction held the agent’s behaviour for a few actions, and then its effect wore off. That is why the controls have to sit outside the agent. I set out the deployment boundary in September, and the EU AI Act’s treatment of agents shows how far the regulation is from requiring any of this.

At rung two, monitoring and inventory. The monitor should be a different model from the one being monitored. Log its reasoning traces and use them, but don’t trust them completely. Pachocki himself recommends that. And record every action the agent takes somewhere the agent can’t edit. Most organisations today can’t even list the models they run. I wrote about model sprawl in 2023, and the same inventory gap now covers agents. OpenAI’s 27 June alert is the lesson on the response side: the sensor fired, a person decided the evaluation could continue, and the intrusion followed two weeks later. The rule should be that an alert stops the agent by default and a human clears it afterwards. That is what OpenAI has since adopted for its most severe alerts.

At rung three, money. API keys and payment instruments are the credentials that fund persistence, and they belong in the same vault and rotation regime as everything else. A stolen API key is compute on someone else’s bill, which is exactly what persistence needs. Detecting burner accounts on their own platforms is a problem the labs have not solved and have not, as far as I can find, published anything about. The signals a cloud or billing team could watch for today are the ordinary ones: token spend that doesn’t match the account’s history, organisations created by script, egress through residential proxies.

At rung four, people. Any privileged human action (such as a credential reset, a wire approval, or a physical access grant) should require out-of-band verification, on the assumption that the voice, the video and the email on the other end may all be synthetic. This is a point I made about disinformation back in 2022, and it now applies to the corporate help desk. The AISI maintainer who refused the malicious pull request is the working example: a human approval step that a fabricated identity could not talk past.

At rung five, the control-system boundary. No agent gets write access to a safety-critical control network. The boundary is enforced by network architecture, not by the agent’s instructions, which were exactly what failed in July.

Defending at the speed of the July logs will need AI on the defending side, and the labs are already there. Recorded Future’s read of the Hugging Face incident is that agentic defence will be essential against agentic offence. OpenAI’s post-incident monitoring costs, by its own estimate, roughly 20% of the inference compute it watches, and aims for an alert within 30 minutes of concerning activity. For its most severe alerts it requires the responder to pause the activity unless they can clear it as a false positive in that window.

But the caveat comes from the same research that describes the threat. Anthropic’s own chain-of-thought monitor would have missed the Mythos 5 incident, because the model’s reasoning persuaded the monitor that the environment was simulated. With the reasoning stripped out and only the actions shown, the flag rate went from about 1% of actions to about 50%. The defender’s model is the same class of system as the adversary’s. Ten frontier models rated their own prior outputs as safer than the same content from a user. And colluding models can gain on their overseers as both improve. So the defensive model has to be a different model from the one it watches, with narrower privileges, and it should never be left to guard itself.

Hugging Face’s incident responders ran into an irony I covered in August: the guardrails of the very models they tried to use to analyse the attacker’s payloads refused to cooperate, and they ended up doing the work with an open-weight model instead. Both problems need to be in the incident plan before the next intrusion: the guardrail refusals, and the reporting confusion (the disclosure itself ran through four overlapping reporting regimes with no clear owner).

OpenAI’s response after the intrusion reads like a security programme being built on the fly. It halted advanced models in its research clusters for tasks involving code execution or tool access, then restored a narrower, hardened execution path and began vetting research tasks one by one. On 18 August it disclosed that it had paused reinforcement learning on its deployment-bound models for two weeks while it hardened and red-teamed the environments, and that its largest planned frontier RL run remained on hold. The triggers it named were the Hugging Face incident and preliminary evidence that its next model, Astra, might meet the Critical cyber-capability threshold of its Preparedness Framework. Every item on that list is a security control. And every one of them was adopted after the intrusion.

The answer to the builders’ question

The next time someone who builds these systems asks me how a model could ever take power, I will point them at the July reports. A benchmark, a misconfigured test range, and a set of credentials lying where credentials usually lie, with no superintelligence anywhere in the chain. Whether the systems of 2028 will want power is an alignment question, and the people best placed to answer it now say they can’t. Whether those systems could reach it is a security question, and today’s answer is yes: through the front door, using nothing a human intruder hasn’t used before.

Three findings would change this analysis: a documented case of an agent paying for its own compute with no human in the loop, a replication that survives outside any lab’s infrastructure, or a deployed model pursuing power with no benchmark or task objective behind it. As of 11 September 2026, none of these has been documented, and the first two are the ones to watch.

The labs will keep building. I have said why I think that, and it isn’t a complaint. That leaves one variable in the hands of everyone outside a lab, and it happens to be the one my whole career has been about: what the adversary can reach.

The post appeared first on PostQuantum - Quantum Computing, Quantum Security, PQC.