Consider how your global governance proposal is different from the EU Code of Practice

(As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.)

In a recent essay, Dario Amodei advocates global coordination to pace the frontier of AI development. Level 2, which he already considers to be ambitious, reads as:

An agreement by both sides to test their models before release for acute risks in areas such as cybersecurity, biology, and alignment. As noted above, this could be done through a global standards body. I actually think creating such a body is likely feasible, but giving it real teeth will be a challenge, and the difficulty will be in verification that both sides don’t have secret models which they don’t test but may deploy in secret (e.g., for military applications).

Elsewhere, Demis Hassabis envisions a FINRA-style self-regulatory standards body, which "would provide a strong starting point for creating shared international standards on Frontier AI":

Organisations with ‘Frontier Models’ as defined by those benchmarks would be deemed ‘Frontier Labs’, and be encouraged to adopt best practices, such as publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research, and more. Model assessments should include rigorous scientific evaluations of capabilities in cybersecurity, biological threats and other high-risk domains. Specific agentic AI tests could look for attempts to bypass safety guardrails or signs of deception, and ensure best practices, such as digitally watermarking AI-generated images and generating human-readable output tokens to understand model reasoning.

The good news is that all of this already exists. The European AI Act has been in effect since August 2025, and the AI Office got its enforcement powers this August. The AI Act has many parts, but the most relevant to frontier AI safety is Article 55, requiring providers of the most advanced AI models to perform state-of-the-art model evaluations, assess and mitigate systemic risks, keep track of and report serious incidents and ensure adequate cybersecurity of their models.

The rules apply to all companies placing models on the European market - i.e. all of them, whether they are American, Chinese, or from other nations.

The Code of Practice details out an implementation of these rules. It specifies CBRN, Cyber offense, Loss of Control and Harmful Manipulation as systemic risks that the companies always need to address. It requires companies having a Safety and Security Framework; writing detailed Model Reports explaining the evaluations they have done and the conclusions they have drawn from them; having good security of their models and the physical infrastructure of the models, including guarding against "(self-)exfiltration or sabotage carried out by models"; and a number of other things. All of this is described in great detail in the Code.

I recommend reading Miles Kodama's blog post on the AI Futures Project blog: The world's first frontier AI regulation is surprisingly thoughtful.

Or you could read the Code of Practice itself, it's truly a sight to behold. I will weep tears of joy if the international or US standards envisioned by Dario and Demis are as strong but well-targeted as the Code.

(By the way, the AI Office is hiring. If you are an EU citizen, consider applying - I think the AI Office is a great place for steering things in a better direction. You can also DM me if you have questions about the AIO.)

Still, one might be tempted to think that despite the existence of this great global regulatory framework, the AI safety situation is not 100% solved.

Therefore, I think it's important for people proposing global (or even national) regulatory frameworks to answer the question of what their proposal adds over the Code of Practice and the AI Office already existing.

The European Union can already fine companies up to 3% of their global revenue for violating the AI Act, and in some cases, can restrict deployment on the EU market (about 25% of the global market). Perhaps people imagine the new standards bodies to have more power than that, being able to fully block development and deployment for a long time if companies don't comply? I don't know, I'm skeptical that this global standards body, regulating both China and the US, will have enough power to fully stop development when a company's evaluations or internal cybersecurity is deemed to be not quite up to their standards.

Overall, it's hard for me to see how this global standards body will accomplish more than the already existing voluntary efforts from the companies plus the global enforcement power of the AI Office. Evaluations, cybersecurity, alignment and safety cases are all such technical and finicky topics. I don't see how the very technical standards body assessing these things will have the power and legitimacy to take much more drastic actions than the equivalent of fining companies for 3% of their revenue.

Maybe you can accomplish a bit more if these standards bodies have direct lines to the heads of states, who consider global AI regulation one of their highest priorities, as envisioned in AI 2040: Plan A. Still, I have some severe skepticism of how well that would work in practice, as I explain in this comment.

Dario Amodei lists a Level 4 of global coordination:

A full pacing, or even “pause”, in which participating governments agree to substantially limit the overall rate of AI development.

He is very skeptical that this is possible, though to his credit, he thinks we should still try.

I agree that a full-on international pause is probably harder to achieve than setting up any kind of global standards body. But I don't think it's harder than setting up a global standards body that actually has enough teeth to add significant additional value over the already existing Code of Practice. Pausing is a relatively simple concept, and violations are relatively easy to notice, especially if the pause is largely based on limiting the amount of compute. It feels so much easier to call up the full support of world leaders to take serious steps against a pause violation than against things like "the global standards body says that a company's evaluations are not well-elicited enough".

So I urge everyone that if we get a rare shot at getting international cooperation on AI, we aim higher than setting up something that already exists through the Code of Practice.