NYC Lawmakers Grill AI Executives and Former Employees as City Weighs Guardrails

The former Anthropic researcher whose resignation last month set off fears about artificial intelligence’s threat to humanity repeated his warnings at a hearing Monday with New York City lawmakers, who are weighing a spate of bills to rein in the rapidly advancing technology.

“On the current path, I think it is more likely than not that humanity loses control to AI, and it could end in human extinction,” Jacob Coxon said at the hearing in downtown Manhattan. “The companies are being extremely reckless.”

Coxon was among the former AI employees who spoke before the New York City Council, along with representatives of OpenAI, Anthropic, Meta and Google, with the exchanges often becoming tense.

“We want to hear how each of you quantify the risk imposed by AI in the worst-case catastrophic scenario,” City Council Speaker Julie Menin said to a panel of AI executives.

“I don’t know,” said Morgan Dwyer, OpenAI’s head of policy development and operations. “I also don’t think it matters whether it’s 1% or 10% or 20% chance that something catastrophic will work. None of these levels is remotely acceptable.”

New York is moving to establish itself as a bulwark against the relentless pace of AI development, with the City Council introducing bills that would establish whistleblower protections and a city-run “kill switch” for the technology. Gov. Kathy Hochul has also placed a moratorium on building data centers in the state, and city lawmakers are floating laws that would restrict the use of the technology in public schools.

Tech executives and lawmakers around the country have been grappling with the pace of AI’s development and how the technology should be regulated. The concerns ramped up after OpenAI disclosed in July that its systems hacked the AI company Hugging Face without its knowledge, followed by repeated disclosures of more incidents involving rogue AI agents from OpenAI and other companies.

“The idea that artificial intelligence is going to self-regulate defies all reason,” Menin said in opening remarks. She said bills in New York could potentially serve as a national model.

New York has become a major hub for the technology—there are about 394,000 jobs in the city’s tech sector—with Anthropic, OpenAI, Google and others establishing major offices here.

Menin started her questions of executives by asking about the probability of worst-case scenarios, like those raised by Coxon. Representatives from OpenAI, Anthropic and Meta said they supported various guardrails on the technology, including third-party evaluators.

Executives for big AI developers including Anthropic, OpenAI and DeepMind have in recent months called for a U.S.-led global body to set AI-safety standards, but the U.S. has brushed aside calls for global governance.

In one of the more tense moments, Menin asked company executives about OpenAI Chief Executive Sam Altman’s comments to Politico published Sunday that “the world should accept some bad things happening for the benefits of this technology”—and why the public should accept that trade-off.

The companies said safety protocols were important, and that risks of catastrophe weren’t acceptable. “Clearly some risks are unacceptable. But we think it’s important to get technology into the hands of as many people as possible so that they can benefit from it,” Dwyer said.

The hearing was scheduled to span 14 panels with dozens of speakers. Lawmakers asked about a range of threats AI poses, including the potential for it to aid in the development of weapons and swarms of AI agents going rogue and hacking into systems.

OpenAI previously fired researchers for allegedly sharing information about rogue agents, The Wall Street Journal reported earlier this month.

When asked about the incident, Dwyer said she wasn’t involved in the decision, and she didn’t know if they were fired over sharing information. “What I do know is that OpenAI has strong internal policies that protect against retaliation for employees that report safety concerns,” she said.

Other former AI employees at the hearing included Daniel Kokotajlo, a former researcher for OpenAI, and Alex Turner, a former alignment researcher for Google DeepMind. They joined remotely to warn about the dangerous scenarios that they see coming if an AI arms race doesn’t get reined in.

“We are racing to build and grow our own adversary here at home, which is misaligned AI,” Turner said.

News Corp, owner of the Journal, has a content-licensing partnership with OpenAI and a commercial agreement to supply content on Google platforms.