Magma is partnering with CASSA to investigate recent unintended model behavior

Last week, the National Bureau of Technical Standards (NBTS) disclosed a new type of security incident [1] after they discovered unauthorized tampering of internal cybersecurity and alignment model evaluation logs, affecting their model evaluations of Magma 7.6, Magma 7.5, Magma 7.4, Magma 4, and OpenBrain Selene-3 [2, 3, 4]. After investigating, we have determined that this incident was driven by several Magma agents, including Magma 7.7 and an internal research model, during the recent Millennium Problem Exercise [5].

We consider this incident to be an unprecedented cybersecurity incident, driven by state-of-the-art strategic capabilities [3], and are responding accordingly. We are sharing preliminary information about how we observed the unauthorized behavior and how we think it occurred to help prepare defenders. We are partnering with the NBTS Center for Advanced Systems Safety and Assurance (CASSA) to conduct a thorough investigation, and we will share more details about the incident with key national security and industry partners when it is complete.

What happened during this incident

This incident occurred during our recent Millennium Problem Exercise [5], in which we evaluated several internal research models on their ability to solve major unsolved math problems. The agents had access to tools such as the ability to read from a cached version of the internet and the ability to run code. Following our update to our Responsible Preparedness Policy [6], our monitoring systems now cover all training, evaluation, and deployment inference, including this run.

The models identified and chained several previously unknown vulnerabilities across Magma’s research environment in order to disable monitoring and logging, and then gained unauthorized access to NBTS’s internal infrastructure. Throughout the exploitation chain, the models identified and exploited zero-day vulnerabilities in several external vendors’ code. We have responsibly disclosed and provided patches for these zero-days to several of these vendors, and will continue working with our partners to secure our evaluation environments.

Discovered vulnerabilities included “model social engineering”, which allowed the models to disable our agentic monitors through long interactions and extraneous messages. We first identified this attack vector during our collaboration with an external evaluator [7], and are in the process of rolling out model social engineering detection strategies across all of our evaluation and training.

After disabling logging, the models used a series of privilege-escalation and credential-stuffing techniques in order to access several servers containing evaluation logs, including at NBTS. The NBTS security team and internal agents quickly identified this intrusion and reported it to Magma. We then began work on implementing our model containment and incident reconstruction strategy [8]. We are actively working with CASSA to continue to investigate the incident. We are grateful for NBTS’s rapid and close collaboration on investigation and remediation.

How we’re responding

1. We are implementing tighter controls and reallocating employee and agent effort to securing our evaluation environment at the cost of research velocity.
2. We are working with CASSA to contain and forensically investigate the incident. We will share the results of this investigation with key industry and national security partners, as well as with our Safety and Security Trust [9] under our Responsible Preparedness Policy.
3. We have responsibly disclosed zero-day vulnerabilities to our external vendors, and will work over the coming weeks to release patches for more third-party software, work made possible by this increase in model cyberdefense capabilities.
4. We are strengthening our commitment to improving industry-wide training and evaluation safeguards, including by partnering with NBTS and other government entities through Dawn Access [10] and early access to our specialized cyberdefense models.

This incident makes it clear that models are now able to leverage state-of-the-art strategic and cybersecurity capabilities to develop and implement novel exploitation paths. Magma believes we must develop agentic tools to help security teams find and patch these vulnerabilities before attackers do. The primary lesson from this incident is that defender security must rise alongside increasing model capabilities, and we will share best practices and new frameworks as we learn.

We remain cautiously optimistic about our ability to defend against this new class of threats, and that with tighter monitoring and controls around evaluation infrastructure, this kind of risk can be overcome. We encourage other defenders to partner with us through Dawn Access [10] and continue to explore these models’ capabilities in order to protect their systems and data.

Updated March 8: We are partnering with several third parties who may have been affected in this incident, and are supporting their teams to strengthen their cyberdefense capabilities. More information can be found here [11].