We Really Are All Going to Die (Just Not Necessarily Because of AI)

Some AI conversations feel not only strangely repetitive, but also unnecessarily catastrophic, with exaggerations that I find somewhat suspicious and, more importantly, not particularly helpful.

Dario Amodei has been warning us for years that extraordinarily powerful AI is coming extraordinarily soon, often without much nuance and with a little of Elon Musk’s taste for exaggeration thrown in. It could transform science, cure diseases, destroy huge numbers of jobs and, if things go badly enough, perhaps destroy us too.

To be fair, AI has changed significantly even since late 2025. Models are better, cheaper (or are they? given that you’ll need the better and more compute intensive models) and more efficient. Agents can do things that seemed implausible a year ago, and some recent security incidents are genuinely worrying.

But Amodei’s basic story hasn’t changed much. Powerful AI is nearly here. It could be wonderful. It could be catastrophic. We must keep building it, but carefully.

There is another factor worth keeping in mind now: Anthropic is preparing for a possible IPO. That doesn’t make its safety concerns completely insincere. But being simultaneously one of the companies building extraordinarily powerful AI and the company warning everyone how extraordinarily powerful AI is creates an interesting set of incentives.

Amodei isn’t alone in calling for more caution. Sam Altman, Demis Hassabis and others (Elon included… although…) at the leading AI companies have also supported greater cooperation, independent evaluation and some form of coordinated restraint. I think there are good ideas there.

But none of that requires believing that there is a “strong chance we could all die.” We can take AI extremely seriously without jumping from increasingly capable and sometimes unpredictable systems to confident predictions of human extinction.

And then along came Jacob Coxon.

After working at OpenAI and Anthropic, the 27-year-old researcher (not one of the main experts on the AI alignment and security matter) left Anthropic warning that the companies were “racing straight to self-improving superintelligence and gambling with our lives.” Then he went on the BBC (and all the main channels in a media tour that probably shows more about him and his incentives than anything else — these guys become quite popular with these approaches):

“There is a strong chance we could all die.”— Jacob Coxon

Jim Morrison had a somewhat better version: “No one here gets out alive.”

Morrison was right, of course (I don’t see AI removing death out of the way, even if it could improve our lives after 60). Coxon may eventually be right too. But if you’re going on the BBC/CNBC/Fox to tell millions of people there’s a strong chance AI could kill all of us in the immediate future, I think you owe us considerably more than the authority of having worked at Anthropic.

Anthropic, meanwhile, hasn’t exactly distanced itself from Coxon. His former boss backed him publicly. Amodei’s latest warnings arrived at almost exactly the same moment. Anthropic reiterated much of the underlying argument.

Maybe that’s simply because they agree. I have no evidence that Coxon’s departure was some orchestrated PR operation, and wouldn’t claim that it was. But as a communications event, it fits Anthropic’s long-running story almost perfectly: even the people building this stuff are becoming frightened enough to leave.

The annoying thing is that I agree with quite a lot of what comes next.

“Yes, there are risks to AI, and good reasons to work much harder on safety. But quoting ‘there is a strong chance we could all die’ is hubristic clickbait.”— Anil Seth

That’s pretty close to where I am.

I want more independent AI research in universities. I want governments to have enough technical expertise to challenge the companies they’re supposed to regulate. I think labs should share information about serious incidents, and independent evaluators should have meaningful access to frontier systems.

Some of Amodei’s latest proposals are sensible. We don’t need to believe there’s a 10 or 20% chance of extinction to justify them.

The harder question is how much confidence we should place in the people making the predictions.

“Dario was already claiming that GPT2 was too dangerous to open source back in 2019. I made fun of them then. Everyone should make fun of them now.”— Yann LeCun

I wouldn’t go that far. GPT-2 being harmless doesn’t tell us that today’s much more capable systems are harmless.

But LeCun raises a good question: where is the prediction ledger?

Amodei has been making extraordinary predictions for years. If AI leaders want governments, companies, students and the rest of us to make decisions based on their forecasts, we should keep those forecasts and revisit them.

Especially because the predictions aren’t only apocalyptic.

Amodei also thinks AI could cure diseases, accelerate science dramatically and perhaps radically extend human life.

“Wow! Let’s pace that shit for sure...”— Martin Casado

There is something fascinating about the duality.

AI might kill us. AI might cure cancer. AI might destroy millions of jobs. AI might create extraordinary abundance.

Perhaps several of those things will happen. But notice what every version of the story has in common: AI is unbelievably powerful, and it is all happening very soon.

That is also a useful story when you’re running one of the world’s most valuable AI companies.

The most cynical interpretation goes considerably further.

“AI is heading for a crash and this cackhanded Terminator narrative is being engineered to save them.”— Michael Spicer

I don’t think we have evidence for that.

There are obvious commercial incentives around the doom narrative. Regulation can favour incumbents. Being the company that understands the danger better than everyone else can be a powerful position. And an IPO gives Anthropic another reason to tell a compelling story about why its technology matters enormously.

But an argument being useful to Anthropic doesn’t mean Amodei doesn’t sincerely believe it. People can be completely sincere and spectacularly wrong.

And there are good arguments for slowing down that don’t involve Terminator at all.

“I think I’m in favor of a slowdown or even pause, not because I think doomsday scenarios are likely, but just because we’ve barely begun applying existing models to all the things we could apply them to.”— Kenton Varda

I like this argument much more.

AI is already consequential enough to justify caution. Cybersecurity is enough. Biological misuse is enough. Labour disruption is enough. The concentration of extraordinary technological power in a handful of companies is enough.

We don’t need extinction to take AI seriously. Which brings me back to Jacob Coxon. Working at Anthropic gives him knowledge most of us don’t have. It doesn’t give him knowledge of the future.

If an AI system can autonomously find serious vulnerabilities, show us. If models are becoming better at deception, measure it. If they substantially lower the barriers to creating biological weapons, governments need to know. But if you tell millions of people there is a “strong chance we could all die,” the evidentiary standard should go up, not down.

AI safety needs less apocalypse and more science. Show us. Measure it. Let independent researchers reproduce it. Tell us what would prove you wrong. And when you make a prediction about what AI will do in six months, two years or five years, write it down.

Then come back and see how you did. Jim Morrison didn’t give us a probability either, but at least he knew he was writing a song.

And then came Donald Trump: “no way we’re slowing AI development”.

(Trump labeled the doomsday warnings and existential risks as the work of unnamed “negative forces” bringing up hypothetical scenarios “that won’t happen.” While he acknowledged the U.S. could implement minor “guardrails,” he firmly stated that artificial intelligence is ultimately “going to be more good than bad, but by a lot.”)

emot | Random Stories is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.