Revenge of the Nerds

The seminal parable of the AI doomster cult is the paperclip maximizer, a fairy tale (specifically, the magic mill Aarne–Thompson type 565) pretending to be a thought experiment. As with most rationalist writing, it has the logical trappings, but is at its core a story of magic, presupposing an omniscient and omnipotent machine no more grounded by the laws of nature and the principles of engineering than the box at the bottom of the ocean making seawater salty.

Most rationalist doomsday scenarios, however, have a much more recent and obvious source: juvenile science fiction. For every existential threat proposed by these people, you can easily throw together a montage of clips from old sci-fi movies without even having to resort to deep cuts (though I do recommend The Satan Bug and its great Jerry Goldsmith score).

The trouble with that approach (other than it being embarrassing) is that science fiction, particularly mass market science fiction, is an extremely poor way of thinking about technology. It leans heavily toward the sensationalistic and apocalyptic. Even in so-called hard science fiction, the demands of storytelling invariably take precedence. The influence of folklore and popular genres like westerns further distracts from serious thinking.

But perhaps the biggest issue is that the appeal of science fiction (even in apocalyptic stories) rests largely in thinking about the possible, while any serious approach to technology starts with thinking about constraints, be they engineering or economic. There are exceptions. Recently, The Expanse did an excellent job building its world, at least the part not involving alien technology, around the constraints of actual physics. On the whole, though, constantly thinking in terms of "what if" instead of "how" is a bad way of approaching real-world problems.

At this point someone in the audience is protesting, "What about all the great predictions science fiction has made over the years?" It is true that science fiction has a great reputation for prescience. It is also true that that reputation is largely undeserved. If you go back all the way to Jules Verne and look at virtually all of the famous cases, you will see that these amazing predictions almost entirely came down to ideas that were in wide circulation at the time of the writing, be they speculations appearing in the popular press or straightforward extrapolations of recent technological developments.

Here too, there are exceptions. Arthur C. Clarke was genuinely ahead of the curve when it came to telecommunications, particularly the technological and societal impact of geosynchronous satellites, and Robert Heinlein deserves credit for his insightful take on AI in The Moon Is a Harsh Mistress, but on the whole, if you go back to the months preceding the writing of some famous "prescient" work of science fiction and look at what was being written about technological breakthroughs and their possible applications, you will see that while the authors may have added a great deal on the literary side, they seldom added much on the engineering side.

This is by no means a damning criticism. No one expects this of other genres (as far as I know, for all their popularity, westerns were not responsible for any major innovations in cattle ranching). Science fiction as a literary art form has nothing to apologize for and it does deserve credit for inspiring an interest in science and technology, but given the extremism of the rationalists and the impact they are having, the extent to which they get their ideas from escapist literature and at best thinly disguised tales of magic is a genuine point of concern.