AI Whistleblowers Say Researchers Are Checking AI Less Frequently

It won’t be any surprise to people with human intelligence that Monday’s New York City Council hearing on the risks of AI was as theatrical as a Broadway show. What was more surprising was what the AI researchers testified about how AI research is evolving, and what that means for their job.

For those who missed it, the city council heard testimony from three AI lab whistleblowers: Jacob Coxon, formerly of Anthropic and OpenAI; Alex Turner, formerly of Google DeepMind; and Daniel Kokotajlo, formerly of OpenAI. Afterwards, they heard from representatives of the AI labs. We’ll get to that lower down.

One of the most interesting parts of the whistleblowers’ testimony related to the AI research process. Coxon and Kokotajlo each said that researchers at the labs have lately been letting AI drive the process of writing code and research. Researchers check AI’s work less frequently, they said.

The labs might not agree with that exact characterization, although their representatives weren’t asked about it at the hearing. But the labs have been the first to say that AI has greatly accelerated the process of AI research. Anthropic even said in a report last month that, as of August, AI was leading more than a quarter of model research and development at the company.

That shift has led some researchers to become jaded. One told me recently that they had left the frontier lab they were at because they felt like they were largely just following the suggestions of the AI and could see their job getting automated in the near future.

This is the context in which people like Coxon have warned about the dangers of AI. Coxon told the city council hearing that it was “more likely than not that humanity loses control to these AI’s, ending in human extinction,” while Turner put his probability of “AI takeover” at one in three. All three whistleblowers described the process of developing AI as “growing” AI, implying that such technology has an organic or human-like quality.

Coxon also made a bold opening statement, saying that “we do not know how to control any AI system yet…we don’t fully control it. We don’t understand its drives or why it does what it does.”

It’s hard to know whether Coxon’s reference to “any AI system” extended to older AI technology—from before generative AI—which has been used for many years in everything from spam classification to recommendations. We’d also imagine that Nvidia’s Jensen Huang and other tech execs who argue that managing such systems is largely an engineering problem would disagree!

Representatives from OpenAI, Anthropic, Google and Meta followed the former researchers to the witness stand. They didn’t get much of a hearing from the council members, who seemed less interested in the answers the representatives did give and more interested in the answers they were unable to give.

Those included answers to questions like the percentage chance that AI would lead to the “worst case catastrophic scenario”; whether their company would bear legal responsibility if their AI were to do something illegal (they should have read my colleague Leo’s story from this weekend!); and whether the companies had insurance for those “catastrophic risk” scenarios.

Morgan Dwyer, OpenAI’s head of policy development and operations, got especially grilled by council members, likely because the ChatGPT-maker has become synonymous with AI-powered breaches following its initial Hugging Face breach and because of reports last week that the company had fired three safety employees for mishandling corporate information.

Council members weren’t impressed with her responses to questions. For instance, when City Council speaker Julie Menin asked Dwyer to estimate the probability of AI causing a catastrophic event, Dwyer responded by saying, “I don’t know. I also don’t think it matters whether it’s a 1% or 10% or 20% chance that something catastrophic will go wrong. None of these levels is remotely acceptable.”

Menin responded that saying, “you don’t know and it doesn’t matter” was a “flippant” answer “at best.”

Here’s what else is going on…

Overheard

Nvidia is fundamentally rethinking the structure of an initiative it announced earlier this summer, the AI Compute Partnership, to provide credit support to AI cloud providers that rent out Nvidia chips, in exchange for a cut of those rental revenues, The Information reported Monday.

OpenAI wrote to the New York City Council last week recommending that the city put safeguards in place to prevent against the risks of advanced AI, City Council speaker Julie Menin said during a press briefing on Monday.

Deals and Debuts

See The Information’s Generative AI Database for an exclusive list of private companies and their investors.

Moonshot AI, which makes the Kimi family of AI models, closed its final private fundraising round at a valuation of roughly $50 billion and is heading toward a Hong Kong initial public offering in the first quarter of 2027 that could raise up to $5 billion, Bloomberg reported.

Etched, which builds AI inference chips, is fielding investment offers that value the company at $40 billion to $50 billion, TechCrunch reported.

DeepSeek, which makes AI language models, is close to raising at least $12 billion in a funding round backed by Tencent and CATL, Bloomberg reported.

JPMorgan is marketing a roughly $5 billion leveraged loan, at a yield near 11%, on behalf of Volta Infrastructure, an AI cloud company building a Norway data center that Anthropic has already signed a six-year compute agreement to use.

Kling AI, a spun-off unit of Chinese video app Kuaishou Technology, has picked banks for a Hong Kong initial public offering that could raise at least $1 billion, Bloomberg reported. Kling, which develops AI video generation models, is aiming for a listing as soon as next year. The company is working with China International Capital Corp., Goldman Sachs and UBS on the share sale, according to the report.

SignSplit, which helps people license personal data for AI, emerged from stealth with a $400 million strategic seed round from W Group at a $1 billion valuation.

Valon, which replaces legacy mortgage-servicing software with AI agents, raised $150 million in a Series D funding round at a $2.3 billion valuation led by new investor Ribbit Capital.

Reactor, which builds infrastructure to run AI world models, added Nvidia's NVentures and Sapphire Ventures to its Series A, bringing its total funding to $74 million.

Namespace Labs, which builds cloud compute for developers, raised $42 million in a Series B funding round led by Scale Venture Partners.

RobCo, a German maker of AI-powered industrial robotic arms, sold $40 million of shares—mostly from current and former employees—in a transaction that doubled its valuation to more than $1 billion, with an investor group including Sequoia and Lightspeed.

Wiremind, which makes AI revenue-management software for transit operators, raised €35 million (about $40 million) in its first-ever funding round led by Seaya Andromeda.

Clockwork.io, whose software keeps AI training jobs running through hardware failures, raised $31 million in a funding round led by Premji Invest, Wing Venture Capital and Seligman Ventures.

OneByZero, which deploys AI systems inside large enterprises, raised $20 million in its first external funding round, a Series A led by Jungle Ventures.

Pandektes, which builds AI-powered legal research infrastructure, raised €13.5 million (about $15.8 million) in a Series A funding round led by Alstin Capital.

SafeWorld, which tests AI-driven robots for safety before deployment, emerged from stealth with a seed funding round of more than $12 million led by Shine Capital and Andreessen Horowitz's Speedrun program.

Ghost, which makes AI-agent hardware, emerged from stealth with an $11 million seed funding round led by Andreessen Horowitz.

UpSmith, which makes AI agents for home-service contractors, raised $10 million in a Series A funding round led by SemperVirens.

Factory, an AI coding startup, received an investment from Menlo Ventures as part of the $5 billion-valuation funding round it announced last month. Menlo declined to disclose the amount.

Huawei agreed to let chip designer Qualcomm license certain patents from Huawei pertaining to AI, 5G, and computing.

Axe Compute completed its acquisition of full ownership of a 2,304-GPU AI computing cluster in Columbus, Georgia from Duos Technologies, Axe Compute announced Monday.

Collibra acquired trail ML, a Munich-based startup whose AI agents automate AI-governance and compliance work, to add continuous compliance assessment and runtime policy enforcement to Collibra's platform.

Teradyne made a strategic investment in Bright Machines, whose software manages AI-infrastructure manufacturing, alongside a plan to integrate Teradyne‘s robotics and test equipment with Bright Machines’ factory platform.

Singapore-based DayOne Data Centers on Monday filed for an initial public offering on the Nasdaq Stock Market, a signal of optimism about the IPO market. A string of companies have postponed plans to go public in recent weeks due to choppy market conditions.

Reflection AI, a 2-year old Nvidia-backed startup, announced its first open-weight model, adding to a collection of U.S. competitors aiming to catch up to the performance of similarly free models from China.

OpenAI said Monday that it will test a new visual ad format that uses images to show how products and services could fit into people’s lives. The ads will first appear while users generate images in ChatGPT, labeled and kept separate from the image being created. OpenAI will also begin watermarking its AI output for users in the EU to comply with the EU AI Act.

Instinct, the AI agent startup, announced that users will now be able to add its agent to chats with friends, which could help in situations like planning travel, buying event tickets and organizing carpools.

Thank you for reading the AI Agenda Newsletter! I’d love your feedback, ideas and tips: [email protected].

If you think someone else might enjoy this newsletter, please pass it forward or they can sign up here.