Notes on “We’re in Triage Mode for AI Safety” by Miles Brundage

This reading was previously included Day 2 of the BlueDot Governance course. In the newest version of the BlueDot Governance course, this reading is now optional. I was sad to see this reading removed because I think it’s a great provocation. However, since it’s no longer part of the core curriculum content, I’m publishing my notes separately from the rest of my day 2 notes.

A short extract:

Miles argues that: “At best, we’ll 80/20 it — mitigating 80% of the risks with 20% of the effort that we would have applied in a world with slower AI progress and an earlier start on serious governance… We will probably not be able to totally avoid all very bad but not quite worst-case scenarios (e.g., a very unjust economic transition that causes a lot of suffering and suicides, rampant AI psychosis and addiction, a poorly planned and rushed transformation of the education system, smaller-scale AI-enabled terror attacks that “only” kill thousands or millions, and stable AI-enabled totalitarianism in at least a few countries for a few decades)“:

Pre-call content:

At risk of taking you down a rabbit hole, I’ll add some context on this reading. Miles was previously a Senior Advisor for AGI Readiness for OpenAI who quit for multiple reasons including feeling he could be less biased outside of industry.

One of the most natural questions to ask is, “Why does Miles believe that we have so little time?” Fortunately, he links to a post that answers this question called Time’s Up for AI Policy (written Dec 2024). He writes:

AI that exceeds human performance in nearly every cognitive domain is almost certain to be built and deployed in the next few years.

The announcement of o3 today makes clear that superhuman coding and math are coming much sooner than many expected, and we have barely begun to think through or prepare for the implications of this

This post links to a Tweet of his around the same time (we can also see his retrospective take from March this year):

I’ll reshare some of my notes on maths progress here:

OpenAI has reported solving the Navier-Stokes Millennium Prize Problem, which would be remarkably above trend if confirmed (there’s a dispute over whether someone else’s work made its way into the training data). However, it feels fair to say that the writing is on the wall (the mathematical community seems to agree, though there are a variety of perspectives on whether we’ll automate the components of a mathematicians job other than just producing proofs and how long that might take). They dropped 372 other results onto the internet and a few days ago, they dropped a few more results.

Aside: Miles has a bet with Gary Marcus on AI progress. In July, Metaculus assigned Gary a 90% chance of winning, but more recently it dropped to 30%.

I’ll share some additional alternate perspectives after the call, but for now I’ll share a quote from Richard Ngo:

I’m concerned that the AI safety community’s focus on short timelines is causing it to operate in a counterproductive scarcity mindset… Focusing on short timeline scenarios is like being a first responder. Most people aren’t temperamentally suited to it. - Full tweet thread

I also have no doubt Eliezer Yudkowsky would be very skeptical of the notion of 80/20-ing:

When I say that alignment is lethally difficult, I am not talking about ideal or perfect goals of ‘provable’ alignment, nor total alignment of superintelligences on exact human values, nor getting AIs to produce satisfactory arguments about moral dilemmas which sorta-reasonable humans disagree about, nor attaining an absolute certainty of an AI not killing everyone. When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, “please don’t disassemble literally everyone with probability roughly 1” is an overly large ask that we are not on course to get.

.. That we have to get a bunch of key stuff right on the first try is where most of the lethality really and ultimately comes from.

Post-call content:

One reason why we might reject triage mode is if we broadly think AI progress is slower than folks at the labs think. We’ve already discussed this, but here’s two more considerations.

• Dwarkesh Patel has argued that timelines are longer than many folk expect because of a lack of continual learning and poor sample efficiency.
• Toby Ord has argued that many people have failed to realise the implications of the shift from pre-training scaling to RL and inference-time scaling. A 100x increase in pre-training is equivalent to a 1,000x increase in inference or a 1,000,000x scale up of RL. He suggests that this effect has been masked since these factors hadn’t been scaled, but he suggests this is set to change with models already utilising significant inference-time compute and RL assigned a significant proportion of training compute (he has written a whole series on AI scaling).

Towards the end of 2024, Steve Newman offered another possibility: what if AI is moving fast, but the “distance” to travel is longer than most people think, since we only become aware of the limitations of models over time?:

• “The real challenges may be things that we can’t easily anticipate right now, weaknesses that we will only start to put our finger on when we observe Claude 5 or o3 performing astonishing feats and yet somehow still not being able to write that tightly-plotted novel.”
• “People usually frame AI progress in terms of a single variable: the rate of progress. They think of AI as either a speedboat, rapidly crossing a lake; or something more like a rowboat, whose tired occupant, after an initial burst of speed, still has a long journey ahead. I think a better analogy would be a speedboat crossing the Pacific Ocean. Current AIs are both astoundingly useful, and limited in many ways. The ocean we call “intelligence” is wider than we can easily appreciate, and will take a long time to cross.”

Here’s a few other reasons why we might reject it:

• Toby Ord talks about broad AI timelines, namely, “The correct epistemic response to the lasting expert disagreement is to have a broad distribution over AI timelines”. He argues that reducing these kinds of timeline distributions to a single number tends to be reductive. I attended a talk he gave on this in Melbourne and I asked him if he had updated based on Mythos, he said that he still thought we should have broad AI timelines.

• Eliezer Yudkowsky appears to agree with Toby: “I make less pretense of knowing exact AI timelines than some. History shows that, while some scientists have correctly called where a tech trajectory will converge, whether later or sooner, there are almost no cases of anyone proclaiming a year and being correct.” Additionally he says: “I looked at “AI 2027” as a title and shook my head about how that was sacrificing credibility come 2027 on the altar of pretending to be a prophet and picking up some short-term gains at the expense of more cooperative actors. I didn’t bother pushing back because I didn’t expect that to have any effect. I have been yelling at people to shut up about trading their stupid little timelines as if they were astrological signs for as long as that’s been a practice (it has now been replaced by trading made-up numbers for p(doom)).”
• Christoph Winter and Charlie Bullock propose “radical optionality” as a strategy for navigating the possibility of transformative AI possibly arriving soon (in the next 10 years) on the basis of us having profound uncertainty over how this will develop and what policy responses are necessary. The term ‘radical’ is operative—unlike other more commone, moderate optionality proposals they advocate investing extraordinary amounts of resources (this reading is now in day 3 of the governance course)