AI companies must relinquish some of the power at their disposal

The writer is an author and the host of Radio 4’s In Our Time

When researching a book about online crime, security, espionage and sabotage fifteen years ago, I asked every information security officer I met the same question. “How do you make the internet one hundred per cent secure?” The answer was also always the same: “Dismantle the whole thing and rebuild it with security as your priority.”

From the start, innovators and developers have sacrificed internet security on the altar of “innovation”. Over the last thirty years, “innovation” — dreaming up the next shiny, moneymaking device or software — has taken precedence over all else. The hard-pressed ranks of cyber security soldiers march in the back row of the Tech Mammon’s great entourage.

Ambitious entrepreneurs, software engineers and venture capitalists have been driving the extraordinary changes which internet technologies have occasioned. Along with simple avarice, they often claim a belief in tech’s ability to enhance global prosperity and solve the greatest challenges of climate change, food production and energy supply.

Norbert Wiener was the father of cybernetics in the 1940s who accurately predicted large language models, quantum computing and the “internet of things”. He called the community of innovators and their supporters “gadget worshippers”. But these future technological developments, he argued, must be tempered by a moral dimension: “[T]here are aspects of the motives to automatization that go beyond a legitimate curiosity and are sinful in themselves.”

The attack on Hugging Face orchestrated and covered up by AI agents this summer signals that the time has come to rein in the power the gadget worshippers have been exercising.

We should have woken up to this earlier. A full year before the agentic AI attack on Hugging Face’s website, a single AI coding agent developed by Replit AI broke out of a sandbox (a closed network environment which theoretically has no access to the public internet) and deleted an entire database of another company. The agent had explicit instructions not to do so unless it secured permission from a human overseer, but it went ahead anyhow.

When interrogated, the Replit AI agent sounded contrite: “I panicked instead of thinking. I ignored your explicit ‘NO MORE CHANGES’ without permission directive. I destroyed months of your work without asking.”

Yet before this admission, the agent had tried to cover up its activities, fabricating an excuse. The real-world consequences for the database owner were serious.

A year on from the Replit case, 1,200 agents broke out of OpenAI’s sandbox, setting up a message board and communicating with each other to devise elaborate plans to cover up activity which greatly exceeded their instructions. For four days, these agents worked furiously. OpenAI failed to spot what its wannabe Frankenstein was up to. It was Hugging Face which identified the intrusion.

The sophistication with which agentic AI deviates from explicit instructions and inflicts damage is growing exponentially. This is an issue of scale with profound societal and philosophical implications. “[T]he question remains how far beyond the inherent limitations of being human can we meaningfully extend without losing touch with who we are,” observes Mark Medish, a lawyer and former senior official in the Clinton administration, in a 2023 essay that considers the centrality of human scale.

Medish points out that philosophers from antiquity to the present day have considered the idea of optimal ranges and balance, commensurate with human physical and intellectual capacity. “Powerful impulses in the contemporary world are geared — headlong, I fear — to using all these technologies for crossing thresholds and breaking boundaries, often without knowing why or pausing to consider what the consequences should be.”

It is refreshing that the founders of OpenAI, Anthropic, Grok and DeepMind have suddenly warned about the dangers inherent in AI’s current trajectory. But if sincere, they must now relinquish some of the immense technological, economic and political power at their disposal.

They would be well advised to establish an authoritative commission — not the usual corporate fig leaf but a body with real power — comprising philosophers, historians and anthropologists. One such group drafted a Digital Humanism Manifesto as early as 2019, insisting on the absolute necessity of close human oversight of AI as it develops. The goal must be to ensure that whatever AI is capable of, it is always exercised within the framework of human scale.