Perhaps unfairly, I decided not to include really knowing Harry Potter plots as relevant expertise

One of the keys to understanding Silicon Valley's profoundly dysfunctional culture is the bizarre antagonism toward expertise. You can find this stated explicitly in places like the techno-optimist manifesto and manifesting itself in the decision to hand countless important positions to men in their early twenties with no relevant expertise, and sometimes no apparent expertise whatsoever.

It is an article of faith in this world that tech bros know more about medicine than doctors, more about finance than bankers, more about education than teachers, more about climate science than meteorologists, and the list goes on. It is no coincidence that the push for hydroxychloroquine started with Elon Musk and Larry Ellison.

This would not be so bad if the press approached the opinions of Musk, Peter Thiel, and Mark Andreessen with an appropriate degree of skepticism, or at least some effort to bring actual experts into the conversation. Unfortunately, the combination of ill-informed but endless tech-sector money combined with credulous coverage has produced a number of huge disconnects between conventional wisdom and the consensus opinions of people who know what they're talking about.

One of the most embarrassing of these gaps is in the humanoid robotics bubble. Billions of dollars are being poured into a design that the large majority of researchers in the field dismissed as lacking serious use cases years ago, and yet you could read a dozen stories of robots running marathons and winning dancing competitions without coming across one quote from an actual roboticist pointing out the limitations of the design.

Perhaps an even larger and more consequential gap exists between the apocalyptic scenarios of the "rationalists" and the people who study these problems for a living.

Bryan Cantrill explains just how bad things have gotten.

From Fool's Expertise:

I listened to the recent Ezra Klein interview with Nvidia CEO Jensen Huang, which seems to be something of a Rorschach test for AI doomerism: if you are inclined to see doom, Jensen sounds dismissive, but if you (like me) do not fear for the future of humanity at the (metaphorical!) hands of a computer program, Jensen’s responses seem very reasonable. (With some exception: does this guy really not know his own zip code?!) But I also think there was a lost opportunity. While Jensen rightfully indicates that existing regulation (e.g., product liability laws) can act to hold the frontier labs accountable for harms caused by their own products (a point made more thoroughly by former FTC chair Lina Khan), he didn’t challenge Klein enough. In particular, Klein repeatedly appealed to the authority conferred by AI expertise; how could Jensen disagree with pioneers like Geoffrey Hinton? While Jensen (rightfully) pointed out Hinton has been wrong in his predictions (e.g., his infamously wrong 2016 prediction that radiology would cease to exist by 2021), he missed an opportunity to explain why Hinton is wrong: it’s not (merely) that Hinton is not seeing the future (or aspects of it) clearly, it’s that Hinton is making claims that far exceed the range of his domain expertise. To take the radiologist prediction (conveniently made long enough ago that it is now unequivocally wrong), we can fairly say that Hinton was wrong because his prediction did not reflect what a radiologist actually does: had Hinton bothered to ask the question instead of making an alarmist admonition (Hinton explicitly said that no further radiologists should be trained in 2016!), he would have learned that practitioners do much more than interpret diagnostic images! Making image interpretation faster or better does not obviate the radiologist — to the contrary, it allows the radiologist to better use their expertise to serve their patients! This pattern of Fool’s Expertise is repeated over and over again among AI doomers: their experience in one kind of system (e.g., LLMs) lends them unwarranted authority in another — authority that goes largely unchecked by people like Klein who don’t delineate between different kinds of expertise. Fool’s Expertise is particularly perilous when discussing catastrophe, which nearly tautologically involves long chains of causation that transcend domains: fully understanding one link does not imply understanding the chain. (This is why when the NTSB investigates an accident, the NTSB Go Team consists of experts across many different domains: they know that a catastrophic accident may involve failures across several different domains, interacting and cascading into broader system failure.) And to say it clearly: to the degree that AI-inflicted doom relies on pathways into the real world, experts on those pathways must be consulted. Is AI a cybersecurity threat? Please talk to cybersecurity experts — who will be quick to point out that the ballyhooed OpenAI escape depended on a pedestrian failure of containment. Are we concerned that AI is somehow going to fashion a novel pathogen? Let’s please consult experts on viral synthesis — and listen to why they think that the fears are misplaced. Or maybe it’s nukes that we’re afraid that AI will get a hold of? Great news: seven decades of living with the possibility of global thermonuclear war gave us not just extensive safeguards but also huge wells of expertise to draw from!