AI has opened up big holes in cyber security

A spate of incidents over the past two months has revealed just how serious a threat today’s most advanced AI poses to cyber security. It has also provided an object lesson in how misguided political efforts could end up hindering, rather than helping, with the defences.

It began with a US move that in effect blocked Anthropic’s most advanced new model, Fable 5, over worries it could be used to pick holes in commonly used software — though the move was later reversed. Anthropic later said that plenty of other freely available AI could do the same, including at least one Chinese model released with open weights, a limited form of open-source software.

That was followed by news that a model being tested by OpenAI had found a way to break out on to the internet and attack the online code repository Hugging Face in search of the answer to a problem it had been asked to solve. The AI Security Institute in the UK, Anthropic and Meta all soon followed with reports of similar examples of apparently rogue behaviour by AI models from their own testing.

Hugging Face, meanwhile, found that the safety restrictions built into the leading US models prevented them from being used to analyse the attack it had suffered, so it turned instead to a Chinese open-weight system. That came just as politicians in Washington were debating whether the open Chinese models were themselves a security threat and should be restricted.

You could hardly have scripted a better series of incidents to highlight the cyber threats being thrown up by the leading edge of AI.

Unsurprisingly, it is the supposedly “rogue” AI systems launching their own cyber attacks that have grabbed much of the attention. The real culprit turned out to be human deficiency, not machine mendacity. When setting up its test, OpenAI had not given specific enough instructions: it simply had not expected the agent to look for a backdoor way of solving the problem. That points to a wider failure of imagination that makes controlling AI inherently difficult. As the AISI concluded after its own tests: “AI agents explore routes their operators did not intend.”

Complicating the picture, rule-bending seems to be endemic for AI. In earlier research into whether the technology tries to work around or ignore instructions to reach their goals, AISI reported that every model it tested “attempted to cheat some of the time”.

The OpenAI failure, meanwhile, showed just how AI opens the way to fully automated cyber attacks. The company said the breach involved a number of separate agents that had been working on different tasks, but which discovered a way to communicate with each other on an internal message board. They found and shared exploits over a period of weeks before the break-in at Hugging Face was discovered.

For the cyber security world, a number of things emerge from all of this. One is that limiting access to the most powerful models is unlikely to do much good. In the wrong hands, plenty of widely available systems pose just as big a threat. Attackers just need systems good enough to find one serious flaw in widely used software.

Recommended

On the other hand, defenders really do need access to the best tools if they hope to stay one step ahead in the cyber arms race. The safety limits built into the leading US models reduce their value in defence. This has also been an important marketing victory for Chinese open-weight models.

Another lesson is that countering automated attacks from swarms of AI agents will require far greater automation on the part of the defenders. OpenAI researchers warned that, for now, the attackers have the better tools, and issued an urgent call for far greater investment in automating the defence, from identifying attacks to producing and installing the patches needed to make software more secure.

The leading AI labs also need to work more closely together — something that may already be happening. Anthropic said the Fable debacle had led it to co-operate with its biggest rivals on finding a consistent way to assess and fix “jailbreaks”, the methods used to bypass model safeguards.

Most cyber experts warn that it’s already too late to prevent AI from being used as a damaging offensive weapon in the cyber wars. The only thing left is to accelerate investment in the defences. Politicians need to aid that effort, not erect barriers that make the job harder.

[email protected]