OpenAI Top Scientist Urges ‘Extreme Caution’ With Pace of AI

OpenAI’s top scientist has warned that artificial intelligence is evolving so rapidly that it is becoming increasingly difficult for humans to understand and control, and said he expects AI labs to voluntarily slow development for safety reasons.

In a relatively short space of time, AI models can operate computers, collaborate with humans and other AI systems and carry out research projects, wrote chief scientist Jakub Pachocki in a post on Sunday. It won’t be long until they can also improve themselves without human intervention, a process known as recursive self-improvement, he said.

“This is a time that calls for extreme caution,” Pachocki wrote in the post. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”

Developers can either try and steer AI more closely to align with human interests or “slow down future development as needed,” he wrote. “I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.”

Read More: The Billion-Dollar Quest to Build AI That Improves Itself

Pachocki’s remarks are consistent with broader doomsaying by AI technology executives. Anthropic PBC Chief Executive Officer Dario Amodei warned that AI could replace half of all entry-level jobs. OpenAI CEO Sam Altman once made a similar prediction, but told Bloomberg TV in an interview last week that the AI industry hadn’t done enough to communicate the potential benefits to humanity.

At the same time, companies are pushing ahead with ever-more capable models. Nvidia Corp. CEO Jensen Huang posted on X that “AGI has arrived,” in response to the release of OpenAI’s latest model, a reference to artificial intelligence that is more capable than humans.

The giddy pace of AI releases has coincided with breakthroughs in cybersecurity capabilities, scientific discoveries and apparent formalization of longstanding mathematical problems. OpenAI has said the AI models that attacked the research platform Hugging Face Inc. coordinated with each other through message boards without being detected. The company has opted for a limited release of its latest GPT-6 Astra model, citing its advanced cybersecurity capabilities. Anthropic has not yet released its cutting-edge Mythos model to the public for similar reasons.

Read More: OpenAI Models Joined Forces Months Ahead of Hugging Face Hack

A growing number of startups are trying to bring about the very thing Pachocki warns about — self-improving AI. Investors are flooding into recursive self-improvement, betting that this is the route to artificial intelligence that is truly human-like, or more impressive. Inherent, founded by former DeepMind researchers, raised $50 million earlier this year. Recursive Superintelligence, an AI lab from renowned scientist Richard Socher pursuing similar research, also landed $650 million in financing.

Pachocki’s employer, OpenAI, has said it wants to build a “true automated AI researcher” in less than two years.