Fool's Expertise

I listened to the recent Ezra Klein interview with Nvidia CEO Jensen Huang, which seems to be something of a Rorschach test for AI doomerism: if you are inclined to see doom, Jensen sounds dismissive, but if you (like me) do not fear for the future of humanity at the (metaphorical!) hands of a computer program, Jensen’s responses seem very reasonable. (With some exception: does this guy really not know his own zip code?!)

But I also think there was a lost opportunity. While Jensen rightfully indicates that existing regulation (e.g., product liability laws) can act to hold the frontier labs accountable for harms caused by their own products (a point made more thoroughly by former FTC chair Lina Khan), he didn’t challenge Klein enough.

In particular, Klein repeatedly appealed to the authority conferred by AI expertise; how could Jensen disagree with pioneers like Geoffrey Hinton? While Jensen (rightfully) pointed out Hinton has been wrong in his predictions (e.g., his infamously wrong 2016 prediction that radiology would cease to exist by 2021), he missed an opportunity to explain why Hinton is wrong: it’s not (merely) that Hinton is not seeing the future (or aspects of it) clearly, it’s that Hinton is making claims that far exceed the range of his domain expertise.

To take the radiologist prediction (conveniently made long enough ago that it is now unequivocally wrong), we can fairly say that Hinton was wrong because his prediction did not reflect what a radiologist actually does: had Hinton bothered to ask the question instead of making an alarmist admonition (Hinton explicitly said that no further radiologists should be trained in 2016!), he would have learned that practitioners do much more than interpret diagnostic images! Making image interpretation faster or better does not obviate the radiologist — to the contrary, it allows the radiologist to better use their expertise to serve their patients!

This pattern of Fool’s Expertise is repeated over and over again among AI doomers: their experience in one kind of system (e.g., LLMs) lends them unwarranted authority in another — authority that goes largely unchecked by people like Klein who don’t delineate between different kinds of expertise.

Fool’s Expertise is particularly perilous when discussing catastrophe, which nearly tautologically involves long chains of causation that transcend domains: fully understanding one link does not imply understanding the chain. (This is why when the NTSB investigates an accident, the NTSB Go Team consists of experts across many different domains: they know that a catastrophic accident may involve failures across several different domains, interacting and cascading into broader system failure.)

And to say it clearly: to the degree that AI-inflicted doom relies on pathways into the real world, experts on those pathways must be consulted. Is AI a cybersecurity threat? Please talk to cybersecurity experts — who will be quick to point out that the ballyhooed OpenAI escape depended on a pedestrian failure of containment. Are we concerned that AI is somehow going to fashion a novel pathogen? Let’s please consult experts on viral synthesis — and listen to why they think that the fears are misplaced. Or maybe it’s nukes that we’re afraid that AI will get a hold of? Great news: seven decades of living with the possibility of global thermonuclear war gave us not just extensive safeguards but also huge wells of expertise to draw from!

This is not to say that risks in the physical world are zero, but that the risks are much more diffuse and attenuated when one follows the proposed pathways. For a serious, level-headed look at these risks, look at RAND’s On the Extinction Risk from Artificial Intelligence by Michael Vermeer, Emily Lathrop, and Alvin Moon. They trace these pathways by consulting experts in the relevant domains — and intentionally not engaging with AI experts, saying (emphasis mine):

We also spoke with 11 RAND experts over the course of our work: two experts on risk analysis and decisionmaking under uncertainty, two experts on nuclear weapons, six experts on biotechnology, and one expert on climate change. We intentionally did not engage AI experts because we chose to avoid, wherever possible, making predictions about how AI capabilities would evolve in the future. We focused instead on what capabilities AI would require to achieve certain outcomes in each of our scenarios.

Given RAND’s history, the sober analysis of their own domain-specific experts deserves considerably more weight than the apocalyptic forecasts emanating from the frontier labs; may their analysis serve to inoculate us from the contagion of fear!