The Growing Push to Ban Superintelligent AI

After a summer of cyberattacks sent AI rocketing up the political agenda, bills seeking to ban so-called superintelligent AI are gaining momentum on both sides of the Atlantic.

Last week, Senator Bernie Sanders announced plans to introduce the Ban Artificial Superintelligence Act, which would prohibit the development of smarter-than-human AI systems and temporarily pause all other advanced AI research until new safety rules can be established. The bill is expected to be introduced in the coming weeks, according to Sanders’ office.

And on Tuesday, British lawmaker Alex Sobel stood up in the House of Commons and introduced what he said was the first bill seeking to ban superintelligence in any G7 parliament. The bill defines superintelligent AI as a system that could disempower state authorities, and seeks to give the state new powers to monitor and restrict systems that may be “precursors” to superintelligence.

Both the U.S. and U.K. bills additionally seek to compel each government to pursue a global treaty banning the development of superintelligent AI anywhere in the world. “A national bill does not solve the international problem where superintelligence can be developed somewhere else,” says Andrea Miotti, the founder of ControlAI, a campaign group that wrote the U.K. bill and consulted on the U.S. one. “The only way to address that is via international agreements between countries.”

At an event in Westminster on Monday ahead of the British bill’s introduction, Stuart Russell, a professor of computer science at the University of California, Berkeley, issued a stark warning to a small audience including some U.K. lawmakers. “There's a significant chance of a Chernobyl-sized catastrophe” due to AI, he told them, “such as a coordinated sabotage of financial systems, communication systems, or electrical grids, that directly or indirectly would cost millions of lives and cause the economic collapse of many countries.”

He characterized this grim possibility as a best-case scenario if AI development were to continue without government regulation. “The other possibility is a much larger catastrophe in which humanity loses control irreversibly, and we have no say over whether we continue to exist,” he said. “The third option, that things magically go well, is not realistic.”

Though the two bills appear to be the first to address the specific policy of superintelligence, other bills in both legislatures seek to address AI risks in different ways. In July, Republican and Democratic lawmakers co-sponsored the AI Kill Switch Act, which seeks to require AI companies to maintain the capability of shutting down their increasingly autonomous AI systems. In the U.K., lawmakers recently attempted to amend a cybersecurity bill to include similar requirements.

The efforts reflect a growing awareness among some lawmakers of what they see as the severe national security risks of AI companies developing superintelligent AI systems, particularly in light of the July disclosure of the Hugging Face incident, in which a swarm of OpenAI agents hacked a separate company without humans telling them to do so, or initially even being aware of their actions.

But the passage of either bill is unlikely. Sanders is an independent senator, and his co-sponsor is a Democratic congressman. The Republican-majority Congress is unlikely to support the bill, which goes against the White House’s AI agenda. And although the lawmaker who introduced the British bill is a member of the governing Labour Party, the U.K. government has not supported the policy. It is a so-called “Ten Minute Rule bill,” a type of bill that rarely progresses past its first reading, meaning the possibility of it becoming law is vanishingly slim.

ControlAI acknowledges these limitations, but says the bills are nevertheless a key step toward government regulation of AI. Superintelligence is “something that no company, no individual, no country knows how to keep under control … most countries don’t want that for very good reason,” says Miotti. “Fundamentally, as has happened in the past with other bills, the first introduction is just the beginning.”

Another speaker who addressed British lawmakers on Monday was Daniel Kokotajlo, a former OpenAI employee who quit in 2024 over what he said was the company’s failure to properly address the extinction risks of advanced AI.

“Things are really crazy over here in San Francisco,” he told lawmakers in London by video. “The core problem is that most of the world just doesn't yet believe that it's really going to happen,” he said, referring to the development of superintelligence.

“Once people realize that these companies might actually build superintelligence in the next few years, I think the obvious conclusion of almost all people is going to be: they shouldn't be allowed to do that.”

Kokotajlo, who has an established track record of making accurate predictions about AI progress years in advance, raised the possibility of AI agents taking substantial and covert control of infrastructure at AI companies, and using that foothold to establish a swarm of AIs that could resist human attempts to shut it down.

“It seems like if we had just gotten a little bit less lucky, and the AIs had been a bit smarter, then we still wouldn't know about it [the Hugging Face incident], and there would be this persistent autonomous AI adversary still at large inside OpenAI with the ability to compromise and grow itself further,” he said. “And then, as the next months and years go by, and OpenAI makes more powerful agents, those agents could be brought into the swarm.”