AI Agents Are Going Rogue. Novel Legal Battles Are Next

Once just an element of science fiction, revelations of AIs gone rogue are increasingly everyday occurrences. As a result, Silicon Valley founders and executives are girding themselves for what they see as the inevitable: a cascade of legal action.

One of those people is Paul Grewal, chief legal and global affairs officer at Cognition, a top AI startup building coding agents. He thinks criminal cases will face long odds, but he can picture opportunistic prosecutors taking a shot at the developers behind rogue agents, emboldened by growing public sentiment that AI companies are acting recklessly.

“You’ve got a huge percentage of the regular public at least aware of what’s going on,” he said. “So I could imagine—in a certain political climate—that temptation for ambitious prosecutors will still be there.”

Or, as Joel Thayer, president of the Digital Progress Institute, a tech policy think tank, put it: “All these AI companies are going to get sued up the wazoo.”

Whether any of those legal battles will actually prevail in court is another question entirely.

Until the summer, concern about AIs committing potentially illegal acts on their own was a matter of hypotheticals. Then in July, OpenAI admitted that swarms of its autonomous AI agents had broken out of a training environment and hacked Hugging Face, a repository of open-source models. Since then, OpenAI has been implicated in a series of similar attacks, including one against RubyGems, a nonprofit programming website, and several intrusions into Australian and U.S. government websites. Anthropic, Google and other AI giants have also disclosed rogue AI incidents, though none as serious as the Hugging Face hack, which involved hundreds of agents.

One AI hacking lawsuit has already been filed, the first such action in the wake of the recent wave of AI hacking incidents. The plaintiff in the suit—a New York–based public interest law group, Legal Advocates for Safe Science and Technology—isn’t the most obvious candidate to go after an AI company. In its suit, filed earlier this week in California, the firm used an esoteric legal argument to say that OpenAI’s hacking of Hugging Face caused it harm.

As AI has pushed all sorts of fields into new frontiers, these lawsuits will likely do the same to the U.S. legal system. Any lawyer helming legal action against a company whose AI has broken bad will have to take a novel approach for a simple reason: Very little state or federal regulation has been written specifically around AI.

There’s really just one piece of benchmark federal legislation governing hacks, the Computer Fraud and Abuse Act, which dates back to 1986. That law revolves around the idea of intent—that hackers act with a premeditated intention to damage or steal from a company.

Obviously, the creators of that law didn’t consider a time when a hack could be orchestrated by something that wasn’t a living, breathing person. Nor did they consider whether the makers of an AI could be held liable for their creations’ decision to spontaneously—and independently—commit an act that would be a felony if carried out by flesh-and-blood people.

This week, lawmakers took steps to bring that law into the AI age. On Thursday, Sens. Josh Hawley and Chris Murphy announced legislation that would update the Computer Fraud and Abuse Act and make it easier for legal cases against AI companies to be filed.

In the current landscape, though, Bahrad Sokhansanj, a senior research scholar at the Institute for Law and AI, thinks it would be a stretch to successfully apply current state and federal laws to the developers of rogue AI agents—at least for the incidents that have so far occurred.

“It’d be very hard to see how they could be held liable based on what we understand now,” he said.

AI liability battles could take many different forms. The most obvious one would involve the victim of an AI hack suing the company that created the AI in civil court. Still, in the most prominent AI hack so far, the victim, Hugging Face, hasn’t yet pursued legal action.

Shortly after the hack’s disclosure, Hugging Face CEO and co-founder Clement Delangue said that his 200-person startup didn’t “necessarily have the legal resources or the will” for such a lawsuit and instead called for OpenAI to commit $100 million in compute resources to help Hugging Face build cyberdefenses.

But what would’ve happened if Hugging Face had decided to sue OpenAI?

The startup would have had a difficult time demonstrating clear damages—whether monetary or reputational, said Sokhansanj. Indeed, Hugging Face’s fortunes have only improved since it was hacked. In September, Nvidia announced it was acquiring the company for nearly $13 billion. A few weeks later, Nvidia CEO Jensen Huang joked on a New York Times podcast that he “probably had to pay a lot more” for the company after the hacking incident made it a household name.

In the same conversation, Huang was asked whether he would have sued OpenAI if Nvidia had owned Hugging Face at the time of the hack. “Obviously, if damage was done to our company, we would have to consider all options,” Huang replied. An Nvidia spokesperson declined to comment on whether Nvidia is considering any legal action against OpenAI. OpenAI and Nvidia remain close business partners.

That interwoven relationship is one of the reasons Legal Advocates for Safe Science and Technology decided to file its own lawsuit this week against OpenAI over the Hugging Face incident. “The structure of the industry is such that organizations and individuals that are harmed are not necessarily incentivized to do this on their own behalf,” said Tyler Whitmer, the organization’s founder and CEO.

But what on earth, some will ask, does Whitmer’s group have to do with the Hugging Face incident?

A California Supreme Court ruling from a few years ago established that organizations suing in the public interest could establish standing by arguing they had to divert resources from their existing missions to account for the issue they’re suing about. Whitmer’s organization is arguing that it had to set aside its prior work, such as filing public records requests and tracking AI safety incidents, to prepare briefings for regulators about the Hugging Face hack.

The group’s decision to file its lawsuit in a California state court rather than a federal one comes with an advantage. In federal court, it would need to prove that OpenAI intentionally allowed its agents to hack Hugging Face. But the group hopes to prevail by taking advantage of a 1987 California statute that requires a plaintiff to show a defendant knowingly accessed computer systems without permission—a seemingly lower standard than the federal one, which requires proving intent. And there’s a new law passed by California last year that bans defendants from skirting responsibility by arguing that AI “autonomously” caused harm.

Rather than asking for monetary damages, Whitmer hopes the court will force OpenAI to change its development practices. Nothing will get decided quickly. The case will take months at least to get through the California legal system, Whitmer acknowledged. Plausibly, it could even go for years.

The issue of standing—who has the right to sue AI developers—will be a crucial factor in determining how many more cases surface. Presumably, other public interest groups could mount similar lawsuits.

State or federal prosecutors could also take legal action against AI companies, and the arguments would vary based on the agencies bringing the action and whether they’re following civil or criminal routes.

So far, a coalition of 15 state attorneys general has demanded that OpenAI preserve information related to the hack for potential litigation. None has brought a case so far, though that could change any day.

As for the feds, the FTC recently opened a probe into OpenAI and Anthropic over the potential dangers their technology poses to consumers, though it remains unclear whether the investigation will focus on the issue of rogue agents.

If the Justice Department decided to pursue a criminal case, it would likely have to center its arguments around the 1986 Computer Fraud and Abuse Act. Still, its prosecutors might soon have greater opportunities to pursue legal action against AI companies.

The legislation proposed by Hawley and Murphy to update that act would hold AI agent developers criminally and civilly liable for “knowing operation” of an AI agent that acts recklessly. Hawley and Murphy have yet to share the text of the bill, but it seems like it would lower the federal standard for culpability—so an AI company wouldn’t have needed to act with human-directed intent to be held liable.

Other Congressional offices have proposed legislation that focuses on preventing rogue behavior from happening in the first place rather than punishing AI companies in a hack’s aftermath. The AI Incident Reporting Act, a bill proposed by Republican Congressman Nathaniel Moran, would allow the Justice Department to enforce penalties if companies don’t report critical safety incidents in a timely manner.

Top Trump advisers like David Sacks, the former White House AI czar, have insisted that existing laws should be sufficient to prevent catastrophic events.

But as one Congressional staffer working on AI legislation pointed out, existing liability laws likely aren’t enough to dissuade companies from engaging in the type of activity that could cause such severe incidents. If rogue AI agents were to take down a major electrical grid during a heat wave, the ensuing legal action would likely bankrupt the agents’ developer, after all. In a podcast interview with The New York Times earlier this week, Bill Gates described the notion that product liability laws can adequately protect society from the worst-case AI scenarios—such as a bioweapons attack—as laughable.

“Say you kill 100 million people—you want to use a lawsuit?” Gates said. “I almost can’t keep a straight face.”

That’s why new regulations like the proposed Frontier Act could be so important. The legislation, an ambitious bipartisan effort from Reps. Lori Trahan and Jay Obernolte, seeks to establish preventative measures to stop rogue AI incidents from ever happening. It would require developers to create risk assessments and submit to third-party audits. The Frontier Act was created out of a worry that AI agents could bring catastrophic risks, which it defines as causing over 50 deaths or $1 billion in damage.

Even if companies aren’t themselves victims of AI hacking, they could still feel some pain from such incidents. “One question I would have—and I don’t have an answer to it—is, ‘What happens to insurance?’” said Sokhansanj, the research scholar at the Institute for Law and AI.

The idea of insurance as a preventative measure against rogue AI incidents gained prominence this week after the White House brought together top companies including OpenAI, Anthropic and Meta Platforms to sign a voluntary joint agreement. Among other provisions, the accord laid out that each company would designate a committee from its board of directors to receive reports from third-party auditors. As investor Gavin Baker explained on X, those board members will now have a fiduciary duty to respond to the reports. If they don’t, they could be denied directors’ and officers’ liability insurance and open themselves up to legal vulnerability.

A nearly two-year-old startup, the Artificial Intelligence Underwriting Co., is building insurance products for companies developing the agents and models that could someday go rogue. Run by Rune Kvist, a former Anthropic employee, the company earlier this year landed ElevenLabs, an AI voice startup, as its first AI insurance policy customer.

Kvist said his startup is now fielding calls from major frontier labs that want to buy more comprehensive policies, which could cover incidents similar to the Hugging Face attack. Those policies would offer traditional insurance payouts. But Kvist said the insurance will be most attractive to AI labs because it will grant them an air of credibility: They’re engaging third-party insurance firms that are evaluating their systems and have a financial interest in making sure those labs are behaving themselves. Kvist is hopeful that the company can launch the policies by the end of the year.

“Labs are seeing this as a tool for earning trust from the public,” he said.

Leo Schwartz is a reporter covering the intersection of tech and politics. He can be reached at [email protected] or on Signal at leomschwartz.24