OpenAI, Anthropic Should Cap Compute for R&D, Say AI Safety Advocates

Ideas are flowing fast and furious on how AI labs can “pace” AI development so safeguards keep up with the technology’s smarts. Last week, OpenAI CEO Sam Altman laid out three scenarios he said have been considered in the industry, including merging all of the AI labs into a single entity to develop new AI safely or creating a global regulatory body like the International Atomic Energy Agency.

In the background, a loose group of safety advocates and researchers have been pushing for AI companies to cap the amount of compute they put towards research and development, a limitation they argue would be particularly effective in slowing AI development.

The researchers are proposing companies commit to dedicating at least some minimum share of their computational resources to safety or monitoring, and another minimum share to running AI for customers, otherwise known as inference. That would limit the share of compute that could be used to develop stronger AI capabilities through R&D, such as training new models.

Limiting the availability of specialized AI chips to research would be a hard pill for the labs to swallow, however. Anthropic CEO Dario Amodei in February named “raw compute” as the top factor that contributed to growth in AI capabilities, while Google DeepMind co-founder Demis Hassabis in April called compute the biggest bottleneck in AI, particularly because research experiments need so much.

What’s more, the size of compute budgets can persuade top researchers to stay or bolt for a competing lab. OpenAI recruiters in past years have advertised greater access to computing resources to attract researchers from rival Google, for instance.

But advocates for the proposal say the idea is worthwhile because it could buy more time for AI companies to put security and safety measures in place, helping prevent more incidents like the rogue agent hacks that have proliferated in past months. In the latest, OpenAI on Tuesday reported instances of its agents weakening their own monitoring systems meant to detect misalignment, or actions not in accord with human desires and values.

“Compute allocation is specific, verifiable, and hard to game,” Thomas Larsen, a researcher at AI forecasting nonprofit AI Futures Project, told The Information. Larsen co-authored “How to Pace The US Frontier,” a framework of how to limit R&D compute, that the nonprofit published in August.

The framework suggests labs could agree to spend 25% of their computing resources on safety, or making sure AI is aligned and won’t commit cyber crimes, 70% on customer inference at 70% and just 5% on R&D. That allocation to R&D would likely be far lower than companies’ current allocation.

The proposal to limit R&D compute is still in the early days of gathering support, but some employees at AI companies are supportive of the proposal, an AI safety advocate who spoke to the employees said.

OpenAI, Anthropic, Google DeepMind and SpaceXSI did not respond to requests for comment.

Companies could limit R&D compute voluntarily by disclosing how they are using their compute, according to the AI Futures Project. Advocates of the proposal say the government or embedded evaluators, such as those working at Anthropic to assess its safety practices and risks, could enforce the caps.

Advocates could also try to bring the proposal to lawmakers for potential AI legislation, building on existing state transparency and auditing bills and proposed bills like the FRONTIER Act, which establishes size-tiered safety obligations on models and would allow the government to restrict models in emergencies.

The advocates for capping the resources used for R&D note that the labs have been addressing their compute plans as they publicize efforts to prevent more breaches. OpenAI in the last few months has shifted about 10% of its total computational resources towards alignment and safety work, we reported last week, though that proportion may have since changed.

At Anthropic, “about 6% of AI R&D compute” was dedicated to safety work in July with “a separate, comparable amount” of compute going to safeguards preventing harmful responses, the company wrote in a September blog post. In fact, Anthropic said that creating a measure of compute usage for different purposes could inform actions such as safety research, or “limits on the share of compute going towards AI research agents.”

However, it’s not guaranteed that definitions of categories such as “safety work” are comparable between AI companies, RAND Europe research leader Henri van Soest said.

One of the goals of the proposal would be to slow down the process of recursive self-improvement, where AI companies use one generation of AI models to make their next generation more capable. Many companies, OpenAI and Meta included, have said their goals include automating research to stay ahead of the competition. Limiting how much compute companies could use for R&D would throw a wrench in that business strategy.

In addition, researchers at AI labs feel immense pressure to continuously innovate because the capabilities of open models, particularly those from Chinese companies, are often able to catch up to the frontier labs within a number of months.

For that reason, some policy researchers say they don’t expect to see support among politicians for a proposal like limiting R&D compute until the U.S. and China both agree to tap the brakes on AI development.

Even if domestic AI companies want to slow down, “once the next improvement from China gets announced, there’s all kinds of political pressure to keep rushing forwards,” van Soest said. “I don’t think you could reasonably have a working pacing deal without an understanding with China.”

Aalok Mehta, director of the nonprofit AI research institute Wadhwani AI Center and former responsible AI policy lead at Google, said he is skeptical the proposal would create much of a slowdown in the pace of AI capabilities growth. That’s because a decrease in compute allocated to R&D could encourage researchers to find ways to train models and conduct research more efficiently.

“Chinese labs are compute constrained, and what that has led to is a lot of innovation in efficiency,” Mehta said.

Here’s what else is going on…

Big Numbers

Anthropic signaled in a draft S-1 viewed by the Wall Street Journal that it could potentially spend $750 billion on compute in the next few years. The AI company had revenue of $16.3 billion and a net loss of over $80 billion in the first half of 2026.

SpaceX reported $14.6 billion in third-quarter revenue on Tuesday, largely driven by growth in its renamed “Super Intelligence” business line, which generated $7.5 billion in revenue from Cursor, Grok Bot, and the company’s burgeoning neocloud business.

Salesforce agreed to acquire Braze, a customer-engagement platform with AI-powered marketing tools, for approximately $5.66 billion.

People on the Move

Mike Clayville, the former Amazon Web Services enterprise sales chief who played a key role in expanding its business with Fortune 500 firms before departing for Stripe six years ago, is back at Amazon as a vice president.

Meta Platforms’ chief global affairs officer, Joel Kaplan, is set to leave the company next year, a Meta spokesperson said, after more than 15 years at the company in which he helped strengthen ties with Republicans including the Trump administration.

Deals and Debuts

See The Information’s Generative AI Database for an exclusive list of private companies and their investors.

A collection of giant insurance firms bought more than $740 million of CoreWeave debt connected to a cloud-computing contract with Meta Platforms in the second quarter, highlighting how insurance money and the AI infrastructure boom are becoming intertwined.

Legora, an AI legal startup, raised a $550 million Series E round at an $8.8 billion post-money valuation led by Atreides Management.

Neolix Technologies, which makes autonomous delivery vehicles, raised $400 million in a Series E funding round led by Boyu Capital.

ASI, which builds AI software for critical infrastructure, raised $325 million in a Series C funding round at a $3 billion valuation, led by Spark Capital.

Pigment, which makes AI-powered business-planning software, raised $250 million in a funding round at a $2.3 billion valuation, led by the Scaleup Europe Fund.

Arda Industries, which builds AI software for manufacturing engineering, raised $160 million in a Series A funding round at a $1.5 billion valuation, led by Altimeter, Accel and Khosla Ventures. The Information first broke the news of the startup here.

Reliable Robotics, which builds autonomous flight systems for aircraft, raised $115 million in new capital consisting of $90 million in equity led by Nimble Partners and $25 million in debt from Silicon Valley Bank.

Nvidia is in talks to invest “hundreds of millions of dollars” directly into Firmus, an Australian AI cloud and data-center operator, after Firmus abruptly pulled its planned $5 billion IPO earlier this month.

Different, which built an AI marketing system for B2B teams, raised $48 million in a seed funding round led by General Catalyst, with participation from Lightspeed Venture Partners.

Baba, which uses AI to improve the Medicare experience, raised $40 million in a Series A funding round led by FirstMark Capital.

Mach9, which develops AI-powered computer-aided design software, raised a $34 million Series A funding round co-led by First Round Capital and Bessemer Venture Partners.

Qualitate, which uses AI for primary market research, raised $25 million in a Series A funding round led by Insight Partners.

AuxoAI, which deploys AI systems for large enterprises, raised $20.7 million in a Series A funding round led by NTTVC.

Firefly, which builds an AI platform for physical safety, raised $20 million in a Series A funding round led by Definition Capital.

Electric Plant Company, which built an AI model for plant health, raised $11.2 million in a seed funding round led by Onto Ventures and Climactic VC.

Elon Musk said Tuesday SpaceX and Tesla have formed special committees to explore a potential merger. This is the first step toward what would be the largest mergers of all-time, bringing together Musk’s sprawling empire of electric cars, rocket launch, satellite and AI businesses.

OpenAI is buying Orchestra Bio, a startup that builds AI agents to help biotech and pharmaceutical companies with drug development, the AI lab announced on Tuesday.

Google announced on Tuesday that TorchTPU, its version of PyTorch, is now available open-source, a sign of how it’s trying to break Nvidia’s stranglehold on the AI industry by making it easier for developers to run AI models on TPUs.

DeepSeek has open-sourced DeepInfer, the in-house framework for running its own models. The move reveals some of the technical secrets behind how the Chinese AI developer achieves high efficiency and low compute costs when running its models at scale.

METR, an AI research nonprofit, reviewed Anthropic’s internal August risk report and concluded they were not confident the report’s methodology would reliably identify risk of catastrophic harm from misalignment.

Compute Labs, a startup founded on the vision of bringing more capital to the market for graphics processing units, has started a matchmaking service for companies looking to buy the chips and lenders willing to lend against them.

AI company Inception launched its model Mercury 3, a diffusion text model or a model that generates whole pieces of content all at once, which it says can generate over 1,000 tokens.

Thank you for reading the AI Agenda Newsletter! I’d love your feedback, ideas and tips: [email protected].

If you think someone else might enjoy this newsletter, please pass it forward or they can sign up here.