Why Focus on Extinction?

In conversations with friends and colleagues about x-risk, I am often asked why I focus on extinction risk – which people find fantastical and distant – when more immediate risks like bioterrorism, gradual disempowerment, and misuse are way easier pills to swallow. Doesn’t that unnecessarily alienate people who would be on your side?

There is certainly a place for talking about prosaic risks, and there are very grave concerns among them, but if I had to choose one message (when you get about five words) it would be about extinction risk.

Strategically, extinction risk is the correct message because Sam Altman doesn’t want to die.

I am not confident Sam Altman cares if there’s a man-made Covid 2. I am not confident Sam Altman would press a button to save math academia if it would impact his bottom line. Data centers are wildly unpopular among the American people; I am not confident this matters if Sam Altman wants more data centers.

I am confident that Sam Altman does not want to die in the next decade.

We do not live in a functioning democracy, and the public’s attention span is short. If we only get one AI risk to shout from the rooftops, it had better be one that the oligarchs will cooperate with us on, if they’re convinced. Cynically, the public perception of AI risk primarily matters as an instrument to pressure those in power; if we pressure them about dangers they won’t actually care about, they’ll write some fluffy tweets and just keep doing what they’re doing.

If A is the set of AI risks that I actually believe in, and B is the set of AI risks that Sam, Dario, Elon, and Xi would slow down for if actually convinced, then “AI will kill us all” is the only element of A ∩ B that I am currently aware of.