Anthropic, OpenAI Executives Urge Oversight of Self-Improving AI

Top executives from Anthropic PBC, OpenAI, Meta Platforms Inc. and Microsoft Corp. are urging policymakers to scrutinize the extent that artificial intelligence systems are improving themselves and take steps to safeguard the technology, adding to calls within the industry for greater oversight of AI.

In a new paper, more than 20 AI leaders and researchers expressed concerns that the emerging practice of using artificial intelligence to help automate the process of developing future models could one day lead to a rapid acceleration in the technology’s progress, a phenomenon the authors call an “intelligence explosion.”

Such a scenario would heighten the risk that AI’s capabilities outpace society’s ability to steer the technology and adapt to it, according to the paper, which was authored by Anthropic co-founder Jack Clark and OpenAI Chief Scientist Jakub Pachocki, as well as AI pioneers Geoffrey Hinton and Yoshua Bengio, among others. The researchers also warned of the potential that “automating AI R&D could weaken human oversight.”

The paper calls for policymakers to gain more visibility into how AI is automating research within the top AI labs as well as to develop mechanisms to “steer and constrain an intelligence explosion,” including by implementing stricter safety measures and oversight of data centers involved in automated AI research and development. The group also calls for policymakers to consider emergency response plans for various AI takeoff scenarios.

Leading AI developers have recently escalated their warnings about the potential for catastrophic risks from future, advanced AI systems, fueled by a series of recent breaches involving AI agents and a sense within some firms that the technology is getting better at improving itself. Anthropic said this month that more than a quarter of its AI R&D work is now led by its Claude chatbot, up from essentially nothing at the start of the year.

“Preliminary evidence suggests that a software-driven intelligence explosion is possible,” according to the paper released Monday. “If one does happen, it could be the most consequential technological development in history.”

Anthropic Chief Executive Officer Dario Amodei issued a lengthy blog post earlier this month calling on the industry to support a broader effort to slow the pace of developing the most advanced models. OpenAI’s Sam Altman and Elon Musk quickly lent their support to Amodei’s plan. Amodei and Altman also urged world leaders to work together on AI during remarks last week before the United Nations Security Council.

“Relative to the stakes, we are not sufficiently prepared,” the researchers said in the paper, before outlining steps policymakers should consider. “Once an intelligence explosion begins, the window for action may close.”

The Wall Street Journal reported on the paper earlier.