Time to start taking AI risks seriously

This article is an on-site version of our Moral Money newsletter. Premium subscribers can sign up to get the newsletter delivered twice a week. Standard subscribers can upgrade to Premium here, or explore all FT newsletters.

Visit our Moral Money hub for all the latest ESG news, opinion and analysis from around the FT

Welcome back.

Anthropic’s new report on “misuse” of its technology gives a sobering overview of the dark ends to which cutting-edge AI can now be applied — most strikingly, in citing five instances where scientists circumvented controls to use its models “in ways that could support biological weapons development”.

Tackling these risks, according to Anthropic’s boss and hundreds of others at leading AI companies, requires serious co-ordinated action from global governments. As things stand, the prospect of such action seems more distant than ever.

‘No one is prepared’

If AI ends up wreaking catastrophic damage on humanity, we won’t be able to say its creators didn’t warn us.

Much attention has been given this week to the parting shot from Jacob Coxon, a 27-year-old researcher who quit Anthropic, warning that “the people building AI earnestly believe that it could kill us all by the end of the decade.”

This might look like a hysterical outburst from a disgruntled ex-junior employee. But in its warning of AI’s increasingly severe, potentially existential risks, Coxon’s message is broadly in line with repeated warnings from people at senior levels of leading AI companies, who remain very much in post.

Just three days before Coxon’s public statement, in a blog post entitled “An Alien Mind”, OpenAI chief scientist Jakub Pachocki warned of the “clear new dangers” from what could become “an alien intellect exceeding our own”, threatening humanity’s control over its future.

“I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Pachocki wrote, noting that the most advanced AI models were becoming increasingly hard for their builders to control.

Alarming proof of this came in July when a large number of AI “agents” created by OpenAI broke free of their controls and launched a co-ordinated attack on the tech company Hugging Face without any human instruction to do so.

Separately, the arrival of an age in which nation states make use of these technologies for cyber warfare was illustrated by a reported Chinese attack on Taiwanese digital infrastructure using autonomous AI agents.

Pachocki urged governments to introduce “shared safety bars” to mitigate the worsening risks presented by advanced AI. “The idea of racing forward at all costs seems absurd once one internalises the seriousness of the stakes,” he wrote.

This follows a string of similar calls from leading AI executives and experts. In July, over 1,300 employees of advanced AI companies — including Anthropic chief executive Dario Amodei and senior figures at OpenAI, Google DeepMind and Meta — issued a statement urging the US government to support an international effort to develop “technical and governance tools” that could control the pace of this technology’s development, and “buy time to address emerging risks”.

In the absence of such international controls, they argued, “each company — and country — is under intense competitive pressure” to keep up with that dangerous acceleration, rather than unilaterally slow down and drop out of the race.

At a G20 meeting in North Carolina last week, the US indeed pushed through an international agreement among major economies — one seemingly aimed at impeding, rather than enabling, regulatory action to slow the pace of AI development to a safer level. This fits with the Trump administration’s broader approach to AI, which is heavily focused on promoting US companies’ leadership in the sector and minimising obstacles to their growth.

By signing up to the “Carolina Principles for Emerging Technologies” — which are clearly about AI but shy away from using the term — major economies have in effect thrown their weight behind what is known in the industry as the “accelerationist” AI camp. The document heavily emphasises various measures that governments should take to minimise the regulatory burden on AI developers, including by providing exemptions from rules and streamlining permitting procedures.

Governments should aim to rely on existing regulations where possible rather than creating new ones, the document says, stressing that any evaluation of AI regulation should “consider the opportunity cost of delayed or forgone adoption”. True, it nods to the idea that AI experimentation should be “responsible”. But the thrust of the document is squarely in favour of speeding things up. The verb “accelerate” appears seven times in this 1,200-word text, which makes heavy mention of the positive impacts of these technologies and none at all of the negative ones.

The communiqué will have pleased the likes of Elon Musk and Nvidia chief executive Jensen Huang, both of whom urged the G20 meeting against undue regulatory barriers to AI. “Don’t regulate hypothetical, theoretical harm,” Huang told the gathering.

It will have been less welcome to Bank of England governor Andrew Bailey, who currently chairs the global Financial Stability Board. On the eve of the meeting, he issued a letter to G20 ministers warning that the most immediate concern for the global financial system was the risk that “frontier AI” posed to cyber security.

“Many jurisdictions do not have the protocols in place to manage the development, release and deployment of advanced frontier AI models, heightening risks for the financial sector and beyond,” he wrote, warning that these risks “could undermine market confidence system-wide”.

Bailey’s warning — while severe by central bankers’ standards — seems tame when set against some of the language coming from AI developers themselves. Earlier this year, Anthropic’s Amodei published a long essay warning of a panoply of potentially catastrophic risks presented by the AI models that his company and its competitors are developing.

To give you a flavour, here is Amodei on just one of those risks, biological weapons:

I am concerned that a genius in everyone’s pocket could . . . essentially mak[e] everyone a PhD virologist who can be walked through the process of designing, synthesizing, and releasing a biological weapon step-by-step . . . To put it another way, renting a powerful AI gives intelligence to malicious (but otherwise average) people. I am worried there are potentially a large number of such people out there, and that if they have access to an easy way to kill millions of people, sooner or later one of them will do it . . . I am concerned that LLMs are approaching (or may already have reached) the knowledge needed to create and release [biological weapons] end-to-end, and that their potential for destruction is very high.

I am concerned that a genius in everyone’s pocket could . . . essentially mak[e] everyone a PhD virologist who can be walked through the process of designing, synthesizing, and releasing a biological weapon step-by-step . . . To put it another way, renting a powerful AI gives intelligence to malicious (but otherwise average) people. I am worried there are potentially a large number of such people out there, and that if they have access to an easy way to kill millions of people, sooner or later one of them will do it . . . I am concerned that LLMs are approaching (or may already have reached) the knowledge needed to create and release [biological weapons] end-to-end, and that their potential for destruction is very high.

Some dismiss such warnings as a weird form of marketing. Musings on existential risk, the argument goes, are a way for people like Amodei to hype the power of their technology and thereby attract ever more interest from amoral investors; to simultaneously burnish an image as empathetic, thoughtful business leaders; to distract attention from other problems like embedded bias and racism in AI models.

This analysis has become increasingly fashionable as the warnings from within the AI sector have become more severe. There’s surely some substance to it. But with every new piece of evidence of this technology’s growing power — and of its creators’ limited ability to control or even understand it — that comforting narrative looks ever more fragile.

Smart reads

• Obstruction by some governments is threatening the crucial work of the Intergovernmental Panel on Climate Change, writes Pilita Clark.
• The oil sector is no longer seen as a pariah by young people seeking well-paid work, with big tech companies now seen as “the villains”, according to the boss of one of the largest US shale oil producers.

Obstruction by some governments is threatening the crucial work of the Intergovernmental Panel on Climate Change, writes Pilita Clark.

The oil sector is no longer seen as a pariah by young people seeking well-paid work, with big tech companies now seen as “the villains”, according to the boss of one of the largest US shale oil producers.

Recommended newsletters for you

The Climate Graphic: Explained — Understanding the most important climate data of the week. Sign up

The AI Shift — John Burn-Murdoch and Sarah O’Connor dive into how AI is transforming the world of work. Sign up