OpenAI’s ‘Pacing 2.0’ Problem: How Smart Is Too Smart?

OpenAI is bound to get a reality check soon, if it hasn’t already, on how difficult it is to “pace” AI technology, let alone getting all the frontier AI developers to agree on how to do so together.

CEO Sam Altman last week walked employees through three proposals the industry has discussed for how to safely develop AI, we reported yesterday. Those options included plans for developing an international organization akin to the International Atomic Energy Agency; merging the frontier labs into a single entity that would develop, test and distribute models; or a scenario, however far-fetched, in which one lab races ahead of the others and literally crushes their GPUs, ending their ability to catch up.

It goes without saying that each of those options has major issues. But we’ll say it anyway.

An IAEA-esque organization would require international cooperation from countries developing state-of-the-art AI, including China, which likely is wary of slowing down development given the hostile feelings firms like Anthropic have expressed towards it.

The second proposal, the “Demis plan,” named after DeepMind cofounder Demis Hassabis, would be similar to CERN, the physics lab jointly controlled by dozens of countries to pursue cutting edge research.

But for now, joint efforts like that might be a tough sell, given that the leading AI developers are for-profit entities, each with different shareholders. OpenAI, Anthropic and Google are currently pursuing a self-regulating body that, while much less ambitious than CERN, have already disagreed on important points, my colleague Leo and I reported last month.

The third option, the Eliezer plan, named after LessWrong creator Eliezer Yudkowsky, is perhaps the most troubling, though people at OpenAI might jokingly say Anthropic is pursuing it.

During Altman’s talk with his employees last week, he also said that OpenAI can’t afford to give up too much ground right now, but within the next year it will need to decide which long-term path to pursue.

Altman refers to this decision as “pacing 2.0.” The current approach, “pacing 1.0,” refers to the continued release of frontier models that have passed certain safety tests, while still disclosing any safety concerns associated with the new models, the people who overheard his comments said.

As part of pacing 2.0, OpenAI will also need to adopt a “speed limit” on recursive self improvement—AI that can perpetually improve itself—or some sort of criteria that will help it decide when it should stop developing or releasing ultra-advanced AI, Altman said.

Developing such criteria would be extremely difficult and somewhat subjective, however. For instance, one might argue that an AI that’s 10 times better than a human scientist is tolerable because it can still be controlled and monitored by existing technology, but not an AI that’s 1,000 times or 10,000 times better than a scientist, which may be able to sidestep those guardrails. But it’s unclear how to even measure such levels of intelligence.

It’s safe to say that the industry debates about safety have just begun, and OpenAI and its rivals have their work cut out for them.

Here’s what else is going on…

AI Deep Dive: Interpretability Could Train Models More Efficiently, Says World Mechanics CEO

Earlier this year, a paper called “Interpreting Physics in Video World Models” tackled a key question about how AI models predict the physical world: do they learn separate representations for physical concepts such as velocity and gravity? Or are those concepts lumped together in the patterns the models learn?

It turns out that these concepts tend to be “highly entangled” in video models, said Sonia Joseph, the lead author of that paper, and the co-founder and CEO of a new startup, World Mechanics, in the latest episode of The Information’s AI Deep Dive.

Joseph argues that separating out these concepts to ensure the model learns a distinct representation for velocity and gravity and so on could help train models with less data and compute. For example, at some point while training an AI model, a researcher could look into the math underlying the model to tell whether it has learned object permanence, and if it has not, add in more training data focused specifically on that capability, she said.

Alternatively, scaling up current world models without new approaches could work, she argued. “It'll just take 20 years and be highly inefficient and consume 20 nuclear reactors of compute.”—Rocket Drew

Big Number

Nscale and Anthropic are in talks for a second compute deal worth roughly $45 billion, on top of the $45 billion, six-year agreement the two companies struck in August for 460 megawatts of Nvidia Vera Rubin chips at Nscale's West Virginia campus.

Overheard

Elon Musk on X to give one of his SpaceX shares to every kid in America next year, and will do more such gifts in the future. The 2027 giveaway would be worth nearly $11 billion, but he hasn’t said how it will work.

A Chinese-speaking hacker used AI models from DeepSeek and Anthropic to compromise the websites of more than 100 companies using around $5,000 worth of tokens, cybersecurity company NorForge said in a report released Monday.

Court Watch

The U.S. Department of Homeland Security ordered all federal civilian agencies to remove Chinese AI company DeepSeek’s models from their IT systems.

Prosecutors allege that the OpenAI employee accused of bugging OpenAI’s office had ties to the Saudi Arabian royal family, according to a Sunday filing from the U.S. Attorney’s office in San Francisco.

Policy Watch

The U.S. has sanctioned a startup and a venture capital firm from China connected to Linwei Ding, a Chinese national and former Google software engineer sentenced to prison last month for stealing Google’s AI-related trade secrets.

Deals and Debuts

See The Information’s Generative AI Database for an exclusive list of private companies and their investors.

Tenstorrent, which builds AI chips, raised $1.5 billion in a Series E funding round at a $6.2 billion valuation, led by Trace Capital and Innovation X.

Otter.AI, a note-taking tool, raised $320 million from General Catalyst’s Customer Value Fund.

Eka Robotics, which develops tech to help robots better pick things up, raised a $150 million Series A round at a $1.45 billion post-money valuation led by Andreessen Horowitz and F-Prime.

Town, which builds AI assistants for businesses, raised $108 million in a Series B funding round at a $1 billion valuation, led by Forerunner Ventures and Menlo Ventures.

Dream Labs, which builds AI brains for robots, raised $90 million in a seed funding round led by B Capital and Kindred Ventures.

Daytona, an AI sandbox company, raised $60 million in Series B funding at a $1 billion valuation led by FirstMark.

MemSilicon, which makes memory chips for AI inference, raised more than 400 million yuan (about $60 million) in a Series A funding round led by Boyuan Capital.

Different, an AI marketing startup, raised $48 million in seed funding led by General Catalyst.

Tolan, which makes an AI companion app, raised $30 million in a funding round from investors including Union Square Ventures, Khosla Ventures and Homebrew.

Kapital, a Latin American digital bank, agreed to acquire Woz, a Y Combinator-backed AI coding startup, for $30 million in cash and stock.

Prism Data, which uses AI to assess lenders’ credit risk, raised $20 million in a Series A funding round led by Story Ventures.

High Lander, which builds software for autonomous drones, raised $16 million in a new funding round backed by EDGE Group, STEP, Protego and Kinetica.

Context.dev, which gives AI agents live web access, raised $9 million in a funding round led by General Catalyst.

XRlabs, which makes AI for surgical operating rooms, raised $8.4 million in a seed funding round led by Playground Global.

Vessla, which builds AI agents for property management, raised $3.3 million (€2.94 million) in a pre-seed funding round led by Seedcamp.

Flae Robotics, which makes an AI hospitality agent, raised more than €1 million (about $1.1 million) in a seed funding round backed by Purple Ventures and Garage Angels.

Doris Labs, which uses AI to organize sales data, raised $600,000 in a pre-seed funding round led by Fuel Ventures.

Arosa AI, which uses AI for real-estate development research, raised $500,000 in a seed funding round led by Taryag Capital.

Amazon launched a public beta run of its TorchNeuron software, a tool which allows users to run PyTorch code on Amazon’s Trainium chips, taking aim at Nvidia’s CUDA platform.

Microsoft released new security tools that continuously assess the risk of AI agents and restrict their access.

MiniMax released its M3.1 Flash model, saying it lowers costs from previous MiniMax models and has improved performance on agentic tasks, coding, computer use, and cybersecurity.

Thank you for reading the AI Agenda Newsletter! I’d love your feedback, ideas and tips: [email protected].

If you think someone else might enjoy this newsletter, please pass it forward or they can sign up here.