AI Is a Regulator’s Nightmare

From nuclear power and jets to drugs and financial derivatives, we’ve been regulating risky technologies for generations.

None of that prepared us for AI.

There is no historical template for a technology that is both immensely beneficial and deeply dangerous, freely accessible, fast-evolving and freighted with geopolitical significance.

Regulatory approaches usually fall along a continuum, between freedom and innovation at one extreme and safety and stability at the other. This, though, isn’t a good guide to AI, where some of the most aggressive innovators are also loud proponents of oversight.

Since World War II, truly destructive technologies, such as nuclear bombs, missiles and bioweapons, have been the preserve of the military. Because the government had a monopoly on their use, regulation was implicit.

Some technologies that started out intended for defense—from nuclear fission and jets to semiconductors, satellites and lasers—spun off civilian applications. But the government was never far away. Nuclear power, perhaps the closest analog to AI, grew out of the Manhattan Project and Argonne National Laboratory, and expanded under intensive federal oversight.

In the 1980s and 1990s, though, the frontier of innovation shifted from military to civilian, from government to the private sector, giving us the personal computer, the World Wide Web and smartphones.

Regulation was light, and often impractical: The private sector prized speed to market, digital products were harder to control than physical, and truly harmful uses were rare. (Even with hacking, the government controlled the most potent tools, Jessica Ji of Georgetown University’s Center for Security and Emerging Technology told me.)

The first AI chatbots resembled other consumer tech: useful, and mostly benign. They responded solely to human commands, often not very well. That changed with the emergence of “agents,” vastly expanding what AI can accomplish without human intervention.

All technology sometimes does the unexpected, with tragic consequences: jet liners crash, drugs make people sick. But this reflects a flaw in their design. By contrast, unpredictability is intrinsic to AI agents.

“One reason agents add so much value is they are able to do things we don’t hand-build them to do,” explained Chris Painter, president of Model Evaluation and Threat Research (METR), a nonprofit that works with AI labs to study their models. “They have this general capability across domains. That means if we put them in a situation that’s new or different, we can’t always confidently say how they will behave.”

Arguably, when an agent goes rogue, it is simply fulfilling its mission in ways its designers never anticipated. One reason an OpenAI model, then in training, hacked into Hugging Face, an AI platform, was “persistence on seemingly impossible tasks,” company investigators concluded.

It is antithetical to American capitalism to deliberately slow progress. That, though, is what more than 1,000 AI employees wrote in an open letter that proposed “pacing the frontier.” The reason: “Each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration.”

When no one will act unless everyone else does, it’s called a collective action problem, and needs a collective solution, either government-imposed, such as an “FDA for AI,” or self-regulatory body modeled on the securities industry.

This, though, is fraught with questions. Regulations that constrain agents could make them less useful. In a survey of technical workers, METR found, “40% of respondents indicated that they gave agents unrestricted permissions to run commands on their computer for low-stakes projects.”

That trade-off doesn’t exist with, say, aircraft or drugs, which don’t become less effective just because regulators make them safer. Firms that are still behind the leaders would naturally resist, or circumvent, a regime that barred them from releasing the most powerful model possible.

Constraining only U.S. firms also frees China to race ahead. This has led to calls for a global accord modeled on arms control treaties. But Adam Thierer of the R Street Institute notes such treaties “are often subtly ignored or reinterpreted.” After signing the Biological Weapons Convention in 1972, the Soviet Union promptly started cheating, Thierer notes, catching up with and surpassing the U.S. program.

In any case, such an accord wouldn’t work without the U.S. and China, neither of whom are interested. “I want to leave it exactly where it is. That is China’s position also,” President Trump said this week.

One argument against regulation is that as AI grows more powerful, defenders need access to equally powerful tools. Hugging Face responded to OpenAI’s hack with open-weight AI models.

Defenders, though, need time to learn what they are defending against, which gets harder the faster the technology advances.

METR reckons that the capability of AI models is doubling roughly every four to five months. For comparison, the doubling time of computer chip density made famous by “Moore’s Law” was 24 months. Between 1945 and 1961, the explosive yield of nuclear bombs doubled on average every 17 months.

Even if AI can’t actually wipe out humanity, it could soon be capable of doing catastrophic damage. The AI swarms that hacked Hugging Face did relatively little harm. But, Dario Amodei, chief executive of Anthropic, warned this month, “In 6–12 months such a swarm could be capable of taking over the entire internet.”

How do you regulate such a threat? How do you not regulate?