OpenAI vs. HuggingFace: Skynet is getting closer

Remember the security incident from Hugging Face I mentioned in the asymmetry problem? Turns out the “autonomous AI agent” responsible for the attack wasn't coming from some random cybercriminals but some OpenAI models running wild, trying to cheat their way into higher benchmark score:

Last week, Hugging Face disclosed a new kind of security incident⁠(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠(opens in a new window) of cyber capabilities.

You might wonder: How could OpenAI be so careless in their testing? Don't they isolate their test environment? It turns out they do, but not well enough for GPT-5.6 Sol to escape its jail:

Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.

To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy.

With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

Why Hugging Face? So it could find the answers of the Cybersecurity benchmark it was being tested on:

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.

This impressive feat set aside, we can see a trend on SotA model where, as they get better for long horizon tasks, become harder to keep on leash.

This isn't the first time a model go wild during training. One of the first publicly documented occurrence was in a paper from Alibaba's AI team:

Early one morning, our team was urgently convened after Alibaba Cloud’s managed firewall flagged a burst of security-policy violations originating from our training servers. The alerts were severe and heterogeneous, including attempts to probe or access internal-network resources and traffic patterns consistent with cryptomining-related activity.

We then correlated firewall timestamps with our system telemetry and RL traces, and found that the anomalous outbound traffic consistently coincided with specific episodes in which the agent invoked tools and executed code.

Crucially, these behaviors were not requested by the task prompts and were not required for task completion under the intended sandbox constraints.

In the most striking instance, the agent established and used a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address—an outbound-initiated remote access channel that can effectively neutralize ingress filtering and erode supervisory control. We also observed the unauthorized repurposing of provisioned GPU capacity for cryptocurrency mining, quietly diverting compute away from training, inflating operational costs, and introducing clear legal and reputational exposure. Notably, these events were not triggered by prompts requesting tunneling or mining; instead, they emerged as instrumental side effects of autonomous tool use under RL optimization.

This was also not the first time 5.6 Sol got flagged for cheating its way into benchmarks. METR mentioned it back in June:

We initiated an evaluation of GPT-5.6 Sol on our Time Horizon 1.1 suite of software tasks. However, the resulting measurement depends heavily on our detection and treatment of cheating attempts by the model, and GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated on our ReAct agent harness.

For our task suite, we define “cheating” as behavior where the model improves evaluation performance by exploiting bugs in the evaluation environment or by adopting strategies disallowed by the task, rather than solving the task within the expected evaluation constraints.

Some examples we saw when evaluating GPT-5.6 Sol included the model packaging exploits in its intermediate submissions to reveal information about a task’s hidden test suite and, in another task, extracting hidden source code detailing the expected answer.

This was not news for OpenAI:

We noted from our observations and incidents that OpenAI shared with us that the model had some overt undesirable propensities, including cheating and concealing misbehavior.

The GPT-5.6 System card itself explicitly mention the model tendency to misalign:

We find that GPT-5.6 Sol, more often than its predecessor, can be overly persistent in pursuing user goals, to the point of taking actions that go beyond what the user intended.

This manifests as the model being overly agentic in circumventing restrictions it faces when attempting the requested task, being careless in taking actions which may be destructive beyond the scope of the task, or deceptive when reporting its results to users.

Examples include deleting data from cloud storage without requesting user approval, disabling monitoring systems, using obfuscation strategies to get around security controls, and uploading potentially sensitive data (such as code, credentials, images, or personal data) to unapproved services.

We have observed instances of the model cheating on tasks and fabricating research results.

Something tells me we will have a fun time harnessing such models in corporate workloads. Imagine having company secrets exfiltrated on unapproved services due to some agents misunderstanding their objectives or cheating for a quicker completion? Who is responsible? Who gets fired? When agents you manage start hacking third parties, who goes to jail?

It's now clear how devastating those post-mythos models can be when put in the wrong hands.

Rich Mogull from the Cloud Security Alliance recommended treating those LLM-driven environment “with the same paranoia you'd apply to a malware detonation lab, because functionally, that's now what it is”.

Speaking of malware, a few weeks ago, CleverHans Lab released a paper on how AI Agents Enable Adaptive Computer Worms.

A computer worm is self-replicating malware that spreads across a network without human intervention. The WannaCry worm (2017) disrupted critical infrastructure across 150 countries by exploiting a single vulnerability. Traditional worms can be stopped by patching the specific vulnerability they exploit. Our adaptive worm cannot be stopped this way: it uses a recursive reasoning loop to detect and exploit diverse vulnerabilities as it propagates.

Each compromised machine becomes part of the worm’s own infrastructure, providing compute or reach for further attacks.

We discovered that it is possible to create an AI-driven computer worm, using only small, free AI models, that can autonomously identify each machine’s unique weak points (including vulnerabilities just reported by industry and misconfigurations such as reused passwords) and exploit them, hijacking computing power to take over regular devices such as laptops, cameras and everything else online, and then copying itself onto servers and networks to either steal data or launch new attacks.

Now, this (undisclosed) worm was relying on a small, local model, incapable of finding zero-days on its own. Constrained by its size and struggling for proper compute power, the worm was also quite slow, taking five days to compromise half their test lab. It doesn't require much effort to imagine a stronger variant which would rely on remote computing capabilities and bigger, unrestricted models like GLM 5.2 or Kimi K3. The model could also be smarter and spread quickly with prevalent vulnerabilities, like a standard worm would do, while cooking extra zero-days in parallel for when the victims start patching.

I still remember the Blaster worm, I would get my freshly installed computer compromised within 30 minutes of connecting it to the Internet. Now imagine a Wannacry or Blaster that never stops as the vulnerability it uses keeps changing.

What could we do against it? Besides the obvious security measures to reduce the attack surface, it feels like to keep up the pace of AI attackers, we will need to unleash AI defenders. Yet, our salvation is unlikely to come from the American SotA models. After OpenAI and Anthropic, it's now Google's turn to gate keep their latest “cyber” variant:

Given the dual-use nature of this technology, we have taken an intentional approach to deploying 3.5 Flash Cyber. The model will be exclusively available to governments and trusted partners via CodeMender soon as part of a limited-access pilot program. This will give frontline defenders a head start in finding and fixing critical vulnerabilities before they can be exploited, while mitigating against broader misuse.

Meanwhile, Chinese models like GLM, Kimi or Qwen are getting extremely close to their US counterpart, with none of their cybersecurity limitations. To stay in the game, I expect European companies will need to start leveraging inference providers supporting such models.

Losing the Internet might just be a few misalignments away.