Wanna bet?

Today’s post is brought to you by my sponsor, Mechanize. They’re hiring junior software engineers at $300K/year base salary. Apply now!

* * *

The Pursuit of Happiness is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Imagine you are 20 years old and offered the following bet:

There will be one spin of a roulette wheel, which contains all the integers from zero to X, where X is not disclosed. If the ball lands on zero you die. Any other outcome wins you $1,000,000 and a pill that prevents cancer and heart disease. Do you take the bet? If I live in Switzerland, probably not. If I live in rural Pakistan or Nigeria, I probably would.

Now suppose the value of X remains unknown, but various roulette experts have opinions. A few think that X is zero—that you are certain to lose. The median expert thinks it’s about 19 (giving you a 95% chance of winning), while some experts believe the value of X is extremely high—implying that you are very likely to win the bet. But you don’t know whether the opinions of these experts are valid. Indeed, you are not even entirely sure that the risk of dying is higher if you play the game than if you choose to sit out the game.

Long-time readers know that I am almost entirely agnostic on the question of AI-risk. To me, LLMs seem like magic. “LLMs predict the next token, one at a time.” I have no idea what that means. Heck, I don’t even understand human intelligence, how could I possibly understand an alien intelligence like LLMs?

On the other hand, I consider myself to be a reasonably rational person, with a Bayesian approach to beliefs about the world. While I have no direct opinion on the question of AI risk, I do have opinions on the sorts of arguments being employed by people with much greater understanding of AI than I have. As an outsider, I find some of the debate to be a bit frustrating. In this post, I won’t offer an opinion about AI, rather I’ll offer some opinions about opinions about AI.

1. Why two camps?

I see a lot of debate between “doomers” and “accelerationists”, even though the majority of experts don’t seem to fall neatly into either camp. So why so much debate over those two positions, while the most popular expert opinion is largely ignored? Or perhaps it is not being ignored, rather it is being mischaracterized? Either way it’s annoying.

Polls show that only a small minority of AI experts think that doom is more likely than not, and only a small percentage believe that AI is almost perfectly safe. The median view seems to be something on the order of a 5% risk of extinction, with the average risk being somewhat higher than the median:

AI researchers, AIImpacts 2022 : give “really bad outcomes (such as human extinction)” a 14% probability, with a median of 5%. 82% believe the alignment problem is important.

You might object that even a 5% risk of extinction is really bad, and many people would group these people in with the doomer position (which I’ll call more than 50% risk of extinction).

That’s fine, but much of the debate takes the doomer position to be that human extinction is very likely, which means that lumping these two positions together can be quite misleading. For instance, if markets expected doom, then interest rates would probably rise sharply. If markets came to expect a 5% risk of doom, then rates would only rise slightly. Any discussion of market expectations needs to discriminate between these two positions. Either a 5% extinction risk is not doom, or if it is doom then current market prices are perfectly consistent with the doomer position.

I do believe that market prices, used appropriately, are the best way of evaluating the likelihood of various future outcomes. Some people will object that the markets failed to predict Covid, even after the pandemic was spreading in China. I agree that, ex post, that event looks pretty bad for market efficiency. And I don’t doubt that some scientists made money by selling the market short in February 2020. But I also believe that people overrate market inefficiency during that period, for two reasons:

First, covid was one of a number of pandemic scares, most of which didn’t end up amounting to much (so far.) I’ve read scary stories about SARS1, Ebola, monkeypox, dengue fever, bird flu and lots of other potential dangers. The market had plausible reasons to believe that fears were overblown.

Second, it was a closer call than most people realize as to whether SARS2 (Covid) was something that could be controlled. China did control it fairly quickly, and China has 1.4 billion people—as many as the entire world in 1880. That fact is too often overlooked. Covid eventually did return to China, but only because the rest of the world failed to control it. And it wasn’t just China; New Zealand and Australia were also able to control the virus, only getting new cases from the outside. If those were the only three countries in the world. Covid would have been extinct by the summer of 2020.

Alternatively, if Covid had been just slightly less infectious, the entire pandemic could have been nipped in the bud. That’s probably what the markets (wrongly) assumed would happen. Unfortunately, it was just barely infectious enough to be unstoppable outside of China. Markets wrongly assumed the US and Europe were just as competent (or authoritarian, take your pick) as China and Australia. We weren’t.

[BTW, I’m NOT suggesting we should have taken the China/Australia approach.]

1. Our strange inability to visualize extinction

I recall a poll of AI experts that suggested they were more concerned about the possible impact of AI on election integrity than they were about extinction risk, even though the median expert was quite worried about extinction risk. In some respects, that seems completely nuts.

On the other hand, I do sort of understand this odd poll result. It’s very hard to think about extinction risk, as the idea is so far removed from our daily life. For instance, there’s been a lot of discussion of the fact that a rising risk of extinction should cause higher interest rates. And an increasing number of people are worried about extinction risk. And interest rates have been rising. So . . .

So . . .

So . . .

Yeah, I know what you are thinking.

I read a lot of financial news, and I recall exactly zero articles in the WSJ, FT, NYT, Economist, etc., that mentioned rising extinction risk in an article on the bond market. I’d go even further. If a reporter mentioned rising extinction risk as a factor explaining rising bond yields, I think they’d be viewed as a sort of lunatic. I might view them as a lunatic. And again, that is despite the fact that polls show that most people are increasingly worried about extinction risk from AI, and rising extinction risk would be expected to cause higher interest rates, and interest rates are rising. So, what gives?

I believe people tend to compartmentalize their thinking. One part of the brain has thinking about grand philosophical narratives, like alien visitation, the existence of God, life after death, the power of prayer, and the end of the world. Another part has thinking about pragmatic issues like what groceries do I need to buy this week. And the two compartments are separate. If your child has cancer, one part of you prays to God and another part seeks out expert medical advice. (Indeed, I feel a bit weird even writing about existential risk in this dispassionate tone. I think I can only do so because one part of my brain doesn’t really believe in it.)

I am not saying that doomerism is some sort of weird religious cult (others do make that claim.) I take doomerism quite seriously. Rather I’m saying that our brains put it into that separate philosophical compartment. That’s why financial reporters don’t mention it in their columns, even though it is something that logically should be offered as an explanation for rising rates (albeit not the only explanation—deficits, credit demand for data centers and inflation worries are also reasonable explanations.)

As an aside, I think market data is far more relevant to the impact of AI on economic growth than to the doomer question. Markets clearly don’t put a high probability on doom, but as noted above the median AI expert sees a 5% risk of extinction, which is still very bad. And a 5% extinction risk would be hard to spot in financial markets. Instead, I believe the most useful financial market data is the fact that the level of interest rates remains near historical norms, casting doubt on the hypothesis that economic growth will accelerate sharply in the near future.

I suppose you could argue that slow expected growth is good news. An expected large acceleration of economic growth is more likely to be associated with AI advances rapid enough to pose existential risks. So perhaps there is some market information after all.

1. Our flawed moral intuitions about extinction

When I first started thinking about this issue, I assumed that a 5% risk of human extinction had an expected cost of 400 million lives (5% of the world’s population.) Later, I wondered if that undersold the risk, as extinction would also deprive all future generations from being able to experience life. But that opens up another can of worms. If someone had killed Genghis Khan, would they have also murdered his 40 million descendants? What about abortion? Birth control? What about Parfit’s “Repugnant Conclusion”?

I don’t have any good answers to the question of how to think about potential lives, other than to note that, at an intuitive level, extinction seems a worse problem than everyone currently alive dying. Everyone alive today presumably will soon be dead, within roughly 125 years. (If not, it will be due to AI.) The consolation that life will go on for future generations seems intuitively important.

1. I suspect that AI bets favor the poor.

Right now, AI is enriching many of the world’s richest people. Nonetheless, I suspect that in the long run the cost/benefit calculation for AI tends to favor the poor. If that seems counterintuitive, consider a technology like cell phones. I was perfectly happy with my 20th century landline, but cell phones brought telecommunications services to billions of peasants in the developing world for the first time. I would not take the bet discussed in the top of this post, risking my life for a 95% chance at a million bucks. But if I were a poor peasant in Africa or South Asia, I absolutely would take that bet. If AI does usher in an era of superabundance, the effects will eventually trickle down to the developing world. In America we’d all become billionaires, and Africans would become millionaires.

Sound crazy? You could argue that the average middle-class American is already richer than medieval kings and queens, at least in most important respects.

To be clear, I’m agnostic on whether AI will be powerful enough to either end the world or end scarcity as an economic problem, but the two possibilities clearly seem to be correlated. If AI ends up being merely a modest boost to technology, then neither the greatest hopes nor the worst fears will be realized. If AI ends up being extremely powerful, then I’d expect the poor to be more willing to take the bet. But the decisions on AI regulation will be made in rich countries, which are already showing signs of increasing risk aversion (although I suspect we’ll proceed anyway.)

1. Gross versus net risk.

This one has received some discussion. I seem to recall Tyler Cowen arguing that AI might reduce certain types of catastrophic risks, such as nuclear war and/or pandemics. Elizer Yudkowsky has a similar view:

And one should not fail to mention—for it also impacts upon existential risk—that Artificial Intelligence could be the powerful solution to other existential risks, and by mistake we will ignore our best hope of survival. The point about underestimating the potential impact of Artificial Intelligence is symmetrical around potential good impacts and potential bad impacts. . . .It looks to me like a successful resolution of Artificial Intelligence should help us considerably in dealing with nanotechnology. I cannot see how nanotechnology would make it easier to develop Friendly AI. If huge nanocomputers make it easier to develop AI without making it easier to solve the particular challenge of Friendliness, that is a negative interaction.

The difference is that Yudkowsky nonetheless sees a much greater downside AI risk than does Cowen, as he believes we needed to solve the alignment problem before developing AI (which we don’t seem to be doing.)

AI could also magnify pre-existing risks. recently suggested that AI might allow a bad actor to create a global pandemic, something I’ve also worried about. But note that this risk exists even in a world without AI.

During the 1980s, I recall thinking about how lucky humanity was that HIV could not be transmitted as easily as the common cold virus. Given its long incubation period, a highly contagious HIV could have infected most of the world’s population before it was identified. Even without AI, a bioterrorist might be able to engineer a catastrophic pandemic.

Now assume that AI allows us to build better defenses against pandemics. In this case, having AI might also allow us to better prepare for future pandemics, perhaps by identifying lots of potential classes of dangerous viruses, and stockpiling vaccines for potential outbreaks. As an analogy, some are suggesting that although AI increases the risk of computer hacking, it may nonetheless by a net plus by increasing our cybersecurity capabilities to an even greater extent.

In general, AI optimists tend to argue that more information is a good thing, and that AI is likely to do more good than harm. Artificial super-intelligence (ASI) might bring us a cancer cure much faster than otherwise. But here is one area where we cannot dodge the difficult problem of valuing future generations. If there is a 10% risk of a non-AI generated pandemic that kills 60% of the world’s population, the expected damage is larger than a 5% risk of total extinction from AI, but only if you ignore future generations.

If you also value future generations, then extinction risk suddenly looms vastly larger, so much so that ASI proponents would probably have to find an argument that not doing AI presents equal extinction risks, and it is not clear if that’s possible. Even nuclear war or an extremely bad pandemic would not kill everyone. In that case, the argument for ASI requires net extinction risks to be lower than the median expert fears—well below 5%.

1. Bayesian reasoning

Bayesians are supposed to start with prior beliefs about the world and then adjust their beliefs as new information comes in. I don’t have even a clue as to how to think about AI extinction risk, so if forced to establish a prior I’d probably just pick something like the median view of the AI experts, even though I suspect their estimates are more like “guesstimates”. Or maybe somewhat less, because of the gross/net risk distinction discussed above.

I do believe that most tech people overestimate the risk of mass job loss from AI, but that’s because economics is one area where I feel I do have some expertise. You might think I should then downgrade their AI extinction risk for the same reason that I downgraded their job loss estimates. But the two issues seems unrelated to me, unless I’m missing something.

Perhaps you could sell me on the idea that STEM-types traditionally overestimate all sorts of risks (overpopulation, global warming, job loss, etc.) because they don’t understand economics and give too little weight to how people and markets respond to changing circumstances. Indeed, in my previous post I argued that most non-economists overestimate the “ruin in nations”. Does that apply to AI risk? I don’t know. I don’t understand AI risk well enough to have a sense of whether those economic considerations are relevant to AI risk. I am reluctant to dismiss AI concerns on “STEMs don’t understand economics” grounds because people like Eliezer Yudkowsky most certainly do understand economics quite well.

1. Probabilistic thinking is unnatural

If the universe is deterministic, then true AI risk is either 0% or 100%. When we use probabilities, it does not imply that events are actually random (although they may be for quantum theoretic reasons), rather it is usually just a reflection of our ignorance.

Go back to the thought experiment at the top of the post. If I guesstimated that X=1000, and that I would have a 99.9% chance at a million dollars, then I might well take the bet. Suppose I do take the bet and win, but afterwards it is revealed that there were only 20 numbers on the roulette wheel. Was it a wise bet? Ex post, it still seems so—but only because I won. Ex ante, it might not have been wise.

If there is a 5% (net) risk of extinction from AI, then the accelerationists are likely to end up being both “right” and “wrong”, ex post. Right in the sense that things will probably turn out well, and wrong in the sense that this risk was probably not worth taking, given those odds and the consequences of failure.

Of course that’s not how people think about things. In our world, people tend to think in absolutes. In the 2020 election, I predicted a very narrow Trump win. The experts predicted a comfortable Biden win, say 4% or 5% in the swing state. In fact, Biden won a very narrow victory, about 1/2% in the swing state. I believe my prediction was more accurate than the experts, but most people would say just the opposite. I got the winner wrong. Who is “right” matters more than who is most rational. Or consider a sports analogy. You predict your team to win by one point when the oddsmakers favor the other team by 20. If the other team wins by one point, who’s prediction was more accurate? It depends on what you care about.

1. Inside and outside views

I often have both an inside view and an outside view. The inside view is my personal opinion, blocking out what others think. My outside view also incorporates the views of others. Thus, if my inside view is that we’ll have 5% inflation, and the market predicts 3% inflation, then my outside view might be that we’ll have perhaps 3.01% inflation. In other words, I’m an EMH guy and I put a lot of weight on market forecasts.

But if people completely relied on market forecasts, then efficient market forecasts could never form in the first place. Someone must gather the independent information required to make sense of the world. Inside views are necessary. Here I’m saying that I have almost no inside view of AI risk, as more than any other subject I can think of I just don’t understand how AI works. Does it make sense to anthropomorphize AI? I don’t know. Would a superintelligent AI be ethical? I don’t know.

In the past, I’ve found that other people view my inside view as my “real opinion.” That’s not how I think about things. But I’ve learned that I’m not a neurotypical person, and there are times where I almost feel like I’m an “anthropologist on Mars”. Thus, I’m often misunderstood.

I watch the AI debate and wonder how so many bright people can be so confident in their views, even as they disagree with each other. After all, they are looking at the same basic set of facts. People tell me that I should have an opinion on AI risk. Really? What did Socrates say about true wisdom? More likely, the actual problem is that you do have an opinion. [“You” doesn’t mean everyone, just most people.]

I haven’t taken any policy positions on these issues, partly because I wouldn’t even know what sort of policies would be appropriate if AI risk were known to be a serious problem. “Regulation”? Fine, but what sort of regulation is likely to work? Aren’t the leading doom people skeptical that regulation can solve the problem? A pause? Fine, but what’s the plan after that? I guess you could argue that during a pause in AI development we will gradually learn more about how to reduce AI risk, but is that true? Never create an AI that is smarter than humans? My 1970s pocket calculator was smarter at multiplication than I was, and modern LLMs are already smarter than me at most things, even most fields of economics.

I do feel that I’m still smarter than Fable in some limited areas of money/macro, although perhaps I’m just fooling myself. In any case, it’s not clear to me what it means for AI progress to stop before AIs are smarter than us, given that they are already smarter in so many respects.

I raise these objections not to indicate that I disagree with any specific proposal to address AI risk, rather I’m trying to explain why I haven’t gone on record favoring any specific policy. But I’m certainly open to suggestions. If the median expert sees a 5% extinction risk, we’d be crazy to not at least give consideration to some possible approaches to reduce that risk. But I honestly don’t know which approach is best.

And I haven’t even discussed further complications: We don’t even know the specific risks we face because we don’t have superintelligence. The point of the paperclip maximizer was that the problem could come from an unexpected direction. And what about China—would they cooperate? And how does regulating AI affect the pace of medical progress? Would AI create a “post-human world” where people are no longer in charge, and how should we think about that outcome?

So many unknown unknowns.

1. Please don’t let me be misunderstood

Elsewhere I’ve argued that there is no such thing as “public opinion”, which is based on the notion that there is no such thing as a unified personal identity. Here’s Bryan Caplan:

In particular, if you frame policies as “Government should do more X,” “Government should do less X,” and “Government should stay the course,” public opinion data usually reveals that the latter is the median position. And the main counterexample, category-specific spending, usually goes away if you add a reminder along the lines of “Remember, more spending requires more taxes.”

Do people actually support more spending on education? Yes and no.

The brain has different components, with different opinions. If I’ve just read an essay on AI by Robin Hanson then I’m less worried about existential risk than if I’ve just read an essay by Eliezer Yudkowsky. I’m not smart enough to refute either argument, but neither represent my “actual view”, which doesn’t exist.

A 2022 survey of AI experts found a surprisingly uniform distribution of existential risk estimates from 0.1% to nearly 100%:

At the same time, most superforecasters had views that fit closer to the expected bell-shaped distribution (although a significant minority had very low risk estimates).

Most superforecasters seem to have the ability to adopt the “outside view”. You can think of superforecasters as being people who don’t know whether Hanson or Yudkowsky are correct, and so they put some weight on both views. The “outside view”. In contrast, AI experts presumably rely more on their inside views, which are more widely dispersed. In addition, superforecasters generally have lower risk estimates than AI experts, perhaps because in the past the “nothing happens” forecasting technique has often proved to be quite effective.

Should I put more weight on the views of subject area experts or superforecasters? I’m not sure. seems to favor expert opinion:

Unlike in my younger days, I am now a certified expert truster. So I don’t feel a need to try to teach myself climatology and look into whether scientists are right when they say the world is getting warmer. Becoming wise means understanding that one of the most useful heuristics one can have is “trust the experts unless there’s something obviously insane about what they’re saying.”

Elsewhere Hanania seems a bit skeptical of the doomer hypothesis.

Should one favor experts on AI or experts on forecasting? This is analogous to the question of the economic impact of climate change—do you favor the views of climate experts or economic experts? In that case, I favor economic experts. But AI is such a weird field that I don’t have much confidence that I know which experts to trust, if any.

To summarize, I don’t “believe” AI extinction risk is 5%. I don’t “agree” with people who make that estimate. Rather, I simply note that this is the reported median estimate of AI experts, and thus a reasonable prior to start with. But I have no reason to assume estimates of 1% or 90% are “wrong”.

I don’t “favor” or “oppose” AI. I am open to policy approaches that boost the present value of expected future utility, but do not know what those policies are. At the moment, virus research is my biggest fear, and AI is something that could accelerate that risk. Or perhaps reduce it. I remain agnostic on AI risk.

Sorry to waste your time on this long “I don’t know”, but I strongly believe the world would be better off if people were less confident in their views.

PS. If you are a conspiracy theorist, please go elsewhere. This blog’s not for you. I’ve personally met lots of experts on both sides, and it’s obvious to me that many if not most of these people have sincerely held views. Not everything in the world is a conspiracy, some opinions are just as they seem. Dean Ball and Alex Tabarrok make this point effectively.

A message from my sponsor, Mechanize:

We’re hiring software engineers to build environments and evals that frontier AI labs use to train coding agents.

To get a better sense of the work we do, you can check out GBA Eval, where we had models build Game Boy Advance emulators from scratch and scored their performance.

Base pay starts at $300K/year for junior software engineers, with more for senior roles, plus equity and performance bonuses. Apply here.

The Pursuit of Happiness is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.