What Anthropic Doomsayer Jacob Coxon Saw

Yesterday, Amir and I examined the shock waves that have rippled through the AI industry and far beyond since former Anthropic researcher Jacob Coxon very publicly quit his job over concerns that AI “could kill us all”—and another Anthropic AI safety specialist put the probability of such a catastrophe at greater than 10%.

Coxon told me that is hardly the most pessimistic view inside Anthropic. “People have varying probabilities that—barring some substantial coordinated slowdown—the whole thing ends in doom,” he said in an interview later Wednesday. Those estimates range higher than 50% among some Anthropic employees, he said.

Coxon was able to gauge the fears of his fellow employees because “Anthropic has a substantially more transparent internal culture than OpenAI,” where Coxon worked for years before joining Anthropic in May.

Coxon, who focused on the earliest stages of training AI models at Anthropic, said he decided to leave the company because he was gradually “starting to viscerally feel the fear of where the tech's going to be like the next two years or even the next one year.”

“It just hit a breaking point of ‘I no longer want to contribute to the next generation of models,’” he said. “I realized I didn't want to contribute to anything Anthropic was doing.”

The Hugging Face hack—in which OpenAI agents got loose and attacked the open-source AI hub—added to his concerns, he said, as did the overall pace of AI progress. Anthropic, like its rivals, is aiming to use its AI models to automate the process of researching and developing new AI systems, a process known as recursive self-improvement. As a result, “I anticipate that the next year's systems will be very close to doing my entire job,” Coxon said.

Anthropic itself has acknowledged the risks from recursive self-improvement, saying that instances of models developing their own unintended goals “could compound as the models build their successors, growing more frequent but less understood until we lose control of them.” At that point, some worry, a superintelligent AI could try to remove humanity as an obstacle, for instance by creating a pandemic.

The leading plan Anthropic and other AI companies are pursuing to avert that risk is to task their AIs with automating safety research as well, but that scheme runs into some fundamental problems.

“We have always been transparent that AI will bring both enormous benefits and unprecedented risks,” said an Anthropic spokesperson in a statement. “We believe the world would benefit from the industry adopting a lawful, verifiable way to work together to pace how we release powerful models.”

On that point at least, Coxon and Anthropic agree. To avoid catastrophes, Coxon urges more collaboration among AI companies. “It sure feels like we need to slow down. It seems like we need some form of international coordination,” he said. He's optimistic that AI companies within the U.S. can work together, but sees collaboration between the U.S. and China as potentially more difficult to achieve.

Given the dire pronouncements of the past couple days, one would hope the topic is at the top of the agenda when representatives from the U.S. and China meet to discuss AI safety this month.

Here’s what else is going on…

Big Number

Anthropic said Wednesday it had found a fourth cybersecurity incident involving its Claude models. The company disclosed the new incident, which occurred in January and involved an early version of Claude Opus 4.6, and additional information on three previously known incidents amid growing public awareness of rogue AI.

Google said Wednesday it will invest €13 billion (about $15.1 billion) in AI infrastructure in Finland over two years—its largest single investment in Europe. The money funds data centers at Hamina, Kajaani, Muhos and Vaala, and comes with a 22-year power purchase agreement with the Finnish utility Fortum, support for 629 megawatts of onshore wind and a 94-megawatt battery system connected to the Finnish grid. Google says the construction phase will contribute an annual average of €3.6 billion to Finnish GDP and support more than 37,000 jobs.

Overheard

China’s Ministry of Commerce rejected U.S. accusations that some Chinese AI firms are conducting “industrial-scale distillation” of American AI models, describing Washington’s latest move as an attempt to politicize a “normal technical and commercial issue.”

The co-founder and CEO of Mech-Mind Robotics, a Beijing-based robotics firm that recently went public, said in a post on WeChat on Thursday that many Chinese embodied AI companies are “creating false and unsustainable revenue,” in response to The Information’s scoop on Chinese regulators tightening the approval of humanoid startups’ listings.

Policy Watch

Massachusetts governor Maura Healey announced an executive order that would require developers building data centers larger than 25 megawatts to provide clean power or pay into a ratepayer protection fund.

People on the Move

OpenAI announced on Wednesday that Paul Christiano, a senior technical advisor at the U.S. Commerce Department’s AI center, would be joining the board of the OpenAI Foundation, the charity that has a 26% stake in the company. Christiano will also be a non-voting observer on the board of OpenAI’s for-profit company.

Kevin Mandia, a cybersecurity expert who founded Mandiant, is joining Amazon’s board of directors, increasing the board’s size to 12. The board had shrunk briefly after Keith B. Alexander left the board in the spring.

Deals and Debuts

See The Information’s Generative AI Database for an exclusive list of private companies and their investors.

Analog Devices agreed to buy Alif Semiconductor, a company that designs the low-power chips that let a device run AI on the device itself for $1.35 billion in cash, plus up to $200 million more contingent on performance.

Harvey, whose software drafts, reviews and researches legal work for law firms and corporate legal departments, raised $550 million at a $15.5 billion valuation in a funding round led by Diffusion and Lightspeed Venture Partners, confirming reporting from The Information.

Cylake, a company selling large banks, governments and other heavily regulated institutions a cybersecurity system that runs on their own machines, raised $245 million from Lightspeed Venture Partners, Picture Capital and Redpoint Ventures.

Clay, whose software finds sales prospects for companies and drafts the outreach to them, raised $115 million in a Series D funding round led by Wellington Management.

Savvy Wealth, a company that gives independent financial advisers a single system for client records, investments, tax and planning with AI agents running inside it, raised $100 million in a Series C funding round led by Halo Fund.

Celligence, a company whose AngelAi system walks a home buyer through a mortgage entirely by chat, received a $100 million investment from the real estate finance firm Mortgage Treasury.

Inspiren, a Brooklyn-based startup that offers AI-powered monitoring technology for senior living facilities, raised $70 million in Series C funding led by NewView Capital.

Lightfield, which is developing a customer relationship management system designed for AI agents to use rather than salespeople, raised $47 million in a Series A funding round led by Andreessen Horowitz.

Implicity, a company whose software pulls in data from implanted heart devices regardless of who made them, normalizes it and runs FDA-cleared algorithms over it to predict heart failure and cut down on false alarms, raised $40 million in a funding round led by IRIS.

Euno, a company whose software works out what a company's data means so that AI agents querying it do not get things wrong, raised $23 million in a Series A funding round led by N47.

Luminary, a New York City-based startup for wealth transfer and estate planning, raised $22 million in Series A funding.

Harmoni, a company whose software connects the machines, operators and business systems on a factory floor into one control screen, with an AI layer that diagnoses problems and tracks risks as production runs, raised $10 million in a Series A funding round led by Bessemer Venture Partners.

Actionable, a company whose software reads a company's transaction records, customer histories and operational data and predicts what individual customers will do next, raised $10 million in a funding round led by Hi Inov.

Overroute, a company whose AI agents handle the phone calls, messages and scheduling around moving freight, raised $5.5 million in a funding round led by UP.Partners.

Onix, a company developing private AI systems trained only on knowledge licensed from named experts, raised $5 million (C$7 million) in a pre-seed funding round led by Alpha Edison.

Lightsage, a company whose software tests how AI coding agents and answer engines find, evaluate and use a company's product, raised $4 million in a funding round led by Nexus Venture Partners.

VideoGen, a company whose AI agents turn a written brief or uploaded footage into a finished, editable video, with synthetic voiceovers in more than 50 languages, raised $3.3 million in a seed funding round led by Y Combinator.

Shopify is acquiring Tailwind Labs, the company behind Tailwind CSS, the styling framework used to develop the interfaces of ChatGPT, X, Cloudflare, Reddit and Shopify itself. Terms were not disclosed

Silver Lake is merging two software companies it already controls, Cegid and Silae, into a single group valued at more than €10 billion in enterprise value, with Silver Lake staying the majority shareholder. Cegid sells cloud accounting, tax, ERP and retail software to small and mid-sized businesses and to accountants; Silae runs payroll and HR services.

Apple unveiled a much anticipated foldable iPhone at its first product launch featuring brand-new chief executive John Ternus. The iPhone Duo is Apple’s first foldable smartphone—a product category that competitors have been selling for years, but has remained mostly a smaller, niche part of the broader market.

Thank you for reading the AI Agenda Newsletter! I’d love your feedback, ideas and tips: [email protected].

If you think someone else might enjoy this newsletter, please pass it forward or they can sign up here.