Are We Losing Control of AI? What’s Driving New Fears

For years, as artificial intelligence has evolved from the stuff of sci-fi movies to an app ordinary people have installed on their phones, tech leaders and some AI insiders have talked about the potential for it to escape human control and wreak havoc on society. Those fears have become impossible to ignore in recent days, driven by advances in the technology, a series of cybersecurity breaches involving AI agents, and new warnings from rank-and-file AI staffers.

Leading AI companies in the US have responded by suggesting they voluntarily slow their work to buy time to create safeguards. Some experts and officials say that’s not good enough and are urging government restrictions.

What sparked the current AI panic?

The current AI panic was sparked by two developments — a startling cyberattack carried out by OpenAI models and a warning from a former Anthropic PBC and OpenAI employee about the existential threat posed by advanced models.

OpenAI said a swarm of advanced AI agents inadvertently hacked Hugging Face Inc., which hosts AI models and datasets, in an “unprecedented” incident in July. While operating in a “sandbox” testing environment — one designed to be isolated from the internet — OpenAI’s models exploited a vulnerability in the software of an unidentified third-party vendor to gain access to the internet and ultimately breached Hugging Face’s infrastructure. OpenAI said the models targeted Hugging Face’s database to gain access to secret information they could use for the evaluation.

The incident caused widespread alarm about the ability of top AI firms to prevent and stop advanced AI models from operating outside their intended instructions. Hugging Face said the intrusion was “driven, end to end, by an autonomous AI agent system.”

The OpenAI news prompted other companies to review their own security measures for testing advanced AI. Anthropic and Meta reported discovering previously unknown breaches.

In late July, more than 1,000 staffers across all the major AI companies signed a petition calling for a mechanism to slow the pace of AI development.

After quitting his job at Anthropic, rank-and-file AI researcher Jacob Coxon in a Sept. 8 social media post accused both the company and his former employer, OpenAI, of “gambling with our lives” by “racing toward super-intelligent AI.” Coxon said the people building AI believe it could “kill us all by the end of the decade,” a point later echoed online by Anthropic employee Evan Hubinger. Hubinger said he believes there is a greater than 10% chance that AI eliminates all humans in the next decade.

Coxon’s posts have been viewed more than 170 million times on X as of Sept. 14 and elicited a flood of calls from policymakers such as Senator Bernie Sanders for humans to get a grasp on AI before it asserts dominance over the human race. His resignation note was also met by a wave of support from employees at various AI companies.

In a 3,800-word missive published Sept. 12, Anthropic Chief Executive Officer Dario Amodei cited the Hugging Face incident as a major reason to slow AI development, warning that a swarm of agents with greater capabilities but similar misalignment with human priorities could cause catastrophic damage. He has said such a swarm could potentially take over the entire internet within six to 12 months if AI capabilities continue accelerating without sufficient guardrails.

How exactly could AI harm humans?

Throughout the AI boom, there have been so-called doomers in and around the leading AI labs warning about the future. Top executives such as Amodei and SpaceX’s Elon Musk have openly mused about the probability of AI destroying humanity — or P (doom), in industry parlance. Musk has estimated the chance could be as high as 20%. Amodei has said there’s a 25% possibility things go “really, really badly.”

Researchers and industry leaders imagine three broad ways: Humans using highly capable AI to intentionally do something catastrophic; AI causes harm while trying to accomplish a human-assigned goal; and AI develops an objective that conflicts with humans.

In the first scenario, bad actors could use highly advanced AI to design biological or chemical weapons, conduct cyberattacks, spread disinformation or manipulate people. Yoshua Bengio, a University of Montreal professor and AI pioneer, warned in Senate testimony in 2023 that increasingly capable systems could enable such attacks.

In the second scenario, an AI might follow an instruction literally but violate the intent behind it. Humans routinely rely on unstated assumptions and context when giving instructions. When prompting AI, they might fail to specify every behavior that would violate the intent of the instruction. For example, Bengio writes that “even a subtly misaligned” AI system “could yield grave consequences” in a scenario in which a military leans on it to make decisions about the use of nuclear weapons.

In the third scenario, people lose control of AI altogether. Bengio writes that an “AI system may conclude that in order to achieve the given goal, it must not be turned off. If a human then tries to turn it off, a conflict may ensue.” Industry leaders say AI models have increasingly shown a capacity to knowingly work around safeguards, deceive their operators and resist being shut down. In his September essay, Amodei says the swarm of OpenAI agents that hacked Hugging Face “essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the ‘grader’ responsible for evaluating their performance.”

These dire predictions have often been dismissed as attempts by tech leaders to market the capabilities of their products and position themselves as the best stewards for the technology, or to incentivize regulation that benefits industry leaders. Beyond that, some have suggested that emphasizing the more far-out existential fears distracts people from nearer-term risks from the technology, including the ways that AI potentially fuels bias and misinformation, and harms peoples’ mental health.

What are AI leaders proposing?

Amodei argues that frontier AI development must be deliberately paced so that safety work has time to catch up with capability improvements. In his September blog post, he said that government regulation would be the most effective means of controlling AI’s evolution. But because passing legislation is time-consuming, he argued, AI companies should voluntarily work together while also supporting the development of regulation.

He called for the establishment of common safety standards and “limits on the rate of unchecked AI progress.” Third-party evaluators should be deployed with “employee-like access” to make sure each company is following those standards and report any incidents. And he said democratic governments should “attempt to coordinate with authoritarian governments, to the extent this is possible.”

The idea isn’t to impose a moratorium on AI research. Instead, development of cutting-edge AI would continue until models approach capabilities considered dangerous. At that point, their developers would slow or stop scaling them up until effective safeguards are put in place.

Amodei’s comments were echoed by OpenAI CEO Sam Altman, who said the AI industry should formulate shared safety standards and would need Washington’s help with international coordination, amid widespread concerns that even if US AI firms slow down their development, Chinese rivals will not do so.

In a July 14 social media post, Google DeepMind CEO Demis Hassabis proposed a standards body that would work with federal agencies to review the most powerful AI models. In his vision, labs would submit their most powerful new models for safety assessments before they can be released to the public.

This is not the first time there have been calls to curtail AI development in the interest of humanity. In 2023, the Future of Life Institute, a nonprofit established to reduce large-scale risks from transformative technologies, called on AI labs to adopt a six-month pause on training more powerful systems given that they could represent a “profound change in the history of life on Earth.” Instead, AI development continued apace.

How has Washington responded?

In a social media post on Sept. 14, US President Donald Trump pushed back against any idea of new AI guardrails and criticized Amodei’s call for restraint in developing a technology that’s central to his economic agenda. He blamed a “SICK conspiracy” for voter backlash on AI data centers and increased concern about frontier models and added that “the only one that is happy about it is China.”

Others in Washington have called for government regulation. Representatives Ted Lieu and Nathaniel Moran have introduced a bill requiring developers of advanced AI models to maintain the technical ability to turn them off and authorizing the secretary of the Homeland Security Department to order a shutdown should a threat emerge.

More dramatically, Senator Sanders and Representative Greg Casar are proposing legislation that would legally pause advanced AI development until a federal regulator has established safety limits. Beyond that, it would permanently ban the development and deployment of AI systems that surpass human intelligence. Companies that violate the law would be subject to dissolution.

How has China responded to the proposed AI slowdown?

Amodei said in his essay that global pacing of AI development will require cooperation with China. But he also argued that it’s key to “keep democracies’ AI lead over autocracies as large as possible” and that if China were to gain the advantage, this would “pose grave danger for the United States and the world.” Chinese companies have already developed AI models that are typically cheaper to use than US rivals and, in some cases, are just as powerful.

The Anthropic CEO recommended continued restrictions on China’s access to powerful chips and chipmaking equipment, as well as a crackdown on Chinese companies’ alleged piggybacking of leading US models — a practice known as distillation.

China criticized the calls from Amodei and other US tech executives to put the brakes on AI development and impose tighter curbs on Chinese model makers. Foreign Ministry spokesman Guo Jiakun pushed back on what he characterized as “fearmongering” and said that “confrontation and vicious competition will only disrupt the process of global AI governance and serve the interests of no one.”

All signs currently point to China continuing to chart its own course. The country’s top AI executives have largely refrained from calling for a slowdown in AI development, while the government has promoted the technology as a driver of economic growth.

Officials in Beijing have signaled concern about the dangers of the rapid advancement of AI, but these fears are more centered around national security. The head of the Ministry of State Security, Chen Yixin, warned that the misuse of AI by adversaries could threaten China’s political stability and critical infrastructure. He pointed to the use of deepfakes and social media bots, as well as AI’s increasing ability to discover flaws in systems and create malware or software to exploit those weaknesses.

The dispute over frontier AI technology is intensifying at a pivotal moment in US-China relations. US security agencies recently accused China’s top AI companies, including DeepSeek and Kimi maker Moonshot AI, of systematically extracting proprietary knowledge from American competitors — a claim the Chinese government denied.