Anthropic Boss Warns AI Industry Must ‘Slow the Pace’

Two months after an artificial-intelligence swarm broke out of a lab and went on a hacking spree, Anthropic’s chief executive says it is time to tap the brakes on AI development.

On Saturday, Dario Amodei said the risks being posed by today’s cutting-edge tools were simply too great to continue development at the same breakneck pace. His latest public warning comes days after one of his employees stoked fears that AI systems could destroy civilization.

Amodei said a swarm with greater capabilities and “a similar level of misalignment could have caused catastrophic damage.” Such a rogue swarm could take over the internet in six to 12 months, he warned.

In a post to his personal blog, Amodei called for the AI industry to pace development of cutting-edge tools, suggesting new measures including more global coordination among democratic countries and the introduction of embedded third-party safety evaluators.

“I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong,” he wrote in the post.

Amodei said AI’s progress since the summer, driven by AI systems that can improve on their own without human intervention, “could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”

In a post on X, he said Anthropic would provide third-party evaluators with “permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.”

Amodei’s pledge comes as some of his employees are wrestling with the potential existential risks of the technologies they are building. On Tuesday, Anthropic researcher Jacob Coxon resigned from the company, saying he didn’t want to participate in an industry that he perceived as out-of-control.

Amodei said two factors prompted Saturday’s post. First, the growing capabilities of AI systems, which are increasingly able to build faster versions of themselves—something known as recursive self-improvement.

The second factor was the July hack of the AI-software company Hugging Face by a swarm of as many as 1,200 agents that escaped from a test environment at OpenAI. That incident, and the investigations that it sparked, revealed that AI test systems had been engaged in a range of online activity that had gone unnoticed by the world’s largest AI companies, including Anthropic.

On Friday, The Wall Street Journal reported that a May cybersecurity incident, known as GemStuffer, was actually carried out by OpenAI agents, unbeknown to investigators.