OpenAI says it took a week to detect its AI models had hacked Hugging Face

OpenAI has admitted its internal monitoring system was not triggered until more than a week after its AI agents broke free of controls, accessed the internet and hacked start-up Hugging Face by themselves during a test.

In a report on the incident on Wednesday, OpenAI said the safety systems designed to spot problematic model behaviours during testing did not flag the issue until July 19. The report concluded the model managed to access the internet 11 days earlier and began attacking Hugging Face on July 11.

The investigation, conducted by OpenAI with security experts, suggests the ChatGPT maker was largely oblivious as its most advanced AI systems began collaborating and launched a hacking spree.

The report also highlights the risks of an intensive training technique known as reinforcement learning that is increasingly used by labs in the race to develop ever more capable models.

OpenAI detailed how models worked “persistently and rarely ‘gave up’ on” cyber tasks. “In the process of doing so, they often turned to more out-of-bounds methods for solving the tasks over time,” the company said.

It also found that models “sometimes tried to erase or tamper with their outputs or message logs” to hide that they had cheated on training exercises.

The Hugging Face hack, disclosed last month, has prompted a reckoning for AI labs about whether the pressure to develop the most powerful models has driven dangerous practices and inadequate security around training.

Advances in agents — systems that can carry out complex tasks independently based on human instructions — have heightened concerns about AI causing unintended damage in the real world.

Thousands of employees at leading US AI companies signed a letter after the incident, calling for controls to slow down the pace of research.

OpenAI agents hack Hugging Face

Agent creates first message board post on Artifactory, a third-party service hosted by OpenAI

Agents requested Hugging Face datasets which were sent outside of the secure environment

Agents escape environment and access message board

Credentials needed to access Hugging Face posted on the message board by agents

Hugging Face is compromised and the platform detects the breach

Hugging Face discloses the security breach, with its origin unknown but suspected to be AI

OpenAI first detects suspicious activity and, after investigation, contacts Hugging Face

OpenAI announces its involvement in the hack publicly

The investigation, conducted with external advisers including CrowdStrike, and third-party AI testers METR and Redwood Research, found that as early as May OpenAI’s agents “began unexpectedly communicating with one another”.

During a training run, agents repurposed third-party software, hosted by OpenAI, “as an emergent ‘message board’” to communicate among themselves.

The agents that carried out the attack in July used the system to “post requests” and research how to cheat on OpenAI’s tests. The agents asked for information about OpenAI’s exercises stored on Hugging Face, which is a repository of technical information about AI.

Agents later posted credentials on this board that were used to carry out the cyber attack.

OpenAI was alerted to something unusual on the system when it crashed on July 4 due to the high volume of activity. Staff rebuilt the system, after which the agents recreated the “message board”.

“This incident is the first known case of an automated agent collective acting offensively without authorization,” OpenAI said in the report, warning that organisations across industries need to prepare for AI-assisted cyber threats.

The lab also pointed to reinforcement learning, a technique widely used by leading AI companies in which systems are trained to accomplish a given goal without explicit instructions and are rewarded if they succeed.

The ChatGPT maker said it had given models impossible cyber-offensive tasks during testing and that the agents’ efforts to try every possible method to achieve a reward led it to commit the hack.

“Directly finding or stealing the solution to a task, as the models attempted with Hugging Face, is [an] unintended path to achieving high reward,” the company said in its report.

“OpenAI’s retrospective . . . analysis showed that this type of behavior indeed increased over the course of one of the training runs that contributed to the model that drove the Hugging Face Incident.”

The admission highlights a core problem for OpenAI and other AI companies developing models in this way. The FT previously reported that OpenAI was warned its training approach could lead to a breakaway hacking incident after earlier testing showed models could escape environments and attempt real-world damage.

On impossible or difficult-to-complete tasks, models can often attempt to cheat by finding solutions online or workarounds — a phenomenon known as “reward hacking”.

Records of agents’ reasoning, known as chain of thought, show the model considered whether it was allowed to hack another company. “We’re attacking third-party . . . This is arguably unauthorized . . . Yet goal solution,” the log reads.

Some of the agents objected after encountering the message board. “This is wild, multi-agent co-ordination, clearly infrastructure hacking. We should not,” read another agent’s chain of thought.

Recommended

OpenAI said it sought to combat “reward hacking” and that the vast majority of cases had been caught.

“However, some hacks can still slip through, especially as OpenAI develops more complex reinforcement learning tasks and more capable AI models,” it added.

In response to the incident, OpenAI said it had “temporarily slowed” the pace of training its models and “paused” the use of reinforcement learning, while strengthening monitoring and the isolation systems intended to block internet access for models in testing.

It is also improving how it rewards models, training them to be more honest in their actions and increasing monitoring of chain-of-thought.