Against SURA

The AI Futures Project distinguishes several different plans for navigating the AI transition, which I'll lump into two broader categories:

Plan A (my combination of their Plan A/S/Other) involves working for a global pause on AI development, hopefully lasting decades. During this pause, we either research preparatory technologies like alignment or human intelligence amplification, or take actions to "fortify" the pause so that it can last indefinitely.

Plan C (my combination of their Plan B/C+/C/sort of D) is for a world where we fail at getting a global pause, and race dynamics continue to dominate. If the leading AI companies are alignment-pilled, they coordinate a mutual pause among themselves, lasting the length of their joint lead over the leading non-alignment-pilled AI company. Then they "burn the lead" researching alignment, and release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them. The exact calculus changes depending on who is in the lead when - is there one alignment-pilled AI company with a six-month lead in front of everyone else? Are there two alignment-pilled AI companies with a joint two-month lead in front of their nearest competitor? Etc. The point is, if it's impossible to stop the race, that six-month or two-month lead is the amount of time they have to prepare, and they can spend it more or less wisely. The point of burning their lead at the last moment is that they can do alignment research with very advanced, right-on-the-brink-of-superintelligence AIs. This is a dangerous plan, and there would be many subtle points they would have to get right, but in a world where nobody can get a global pause in time, it's hard to think of a better one.

Plan A benefits from raising mass awareness, educating politicians, and preparing the verification technology that could enforce a pause treaty. Plan C benefits from alignment-pilling companies, having the alignment-pilled companies be as far in the lead as possible, making sure the alignment-pilled companies will burn their lead when the time comes, and giving them as good alignment teams as possible (so that burning the lead produces good results).

From the beginning of AI safety as a community to about 2023, the conventional wisdom was that our chances of Plan A were slim, and we should be preparing for Plan C. From about 2023 onward, as the general population woke up to the implications of ChatGPT, we rightly concluded that the chance of getting Plan A was better than expected, and we should invest more resources into trying to make it work. During both periods, the natural strategy was to work as hard as we can to get Plan A, where chances of survival are highest, but to be prepared to execute Plan C competently as a backup if that didn't pan out.

Now I think the pendulum has swung too far. New members of the community are unaware that Plan C was ever considered a possibility, or think it's some kind of accelerationist plot to distract us from Plan A. Older members heap scorn on it and act like everybody knows it's been debunked. Most concerningly, people propose burning our chances of executing Plan C in exchange for infinitesimal increased chances of getting Plan A, or even for no increased chance of Plan A at all.

I'll lump these ideas together under the heading of "SURA" - the idea that the only ethical course is for people/companies/countries to Stop Unilaterally + Raise Awareness, with no plan for what happens if race dynamics continue anyway. I interpret the planks of SURA's platform as:

Individual alignment-pilled AI companies are morally obligated to stop unilaterally, regardless of whether they expect race dynamics to continue in their absence.
Individual alignment-pilled employees at these companies should quit, regardless of whether they're in capabilities or alignment roles, and regardless of whether they think their company is morally better than its competitors.
Individual alignment-pilled countries (for example, a hypothetical future alignment-pilled US) should ban or heavily restrict AI progress unilaterally, regardless of whether they expect other countries (for example, China) to continue racing in their absence.

All of these planks are net negative in a world where we end up needing Plan C:

If we demand that leading alignment-pilled companies quit the race unilaterally, then we ensure that non-alignment-pilled companies end up in the lead.
If we demand that alignment-pilled company employees quit their jobs, this makes companies less alignment-pilled and less prepared to research alignment if they need to. It also is likely to decrease the lead that alignment-pilled companies have over the nearest non-alignment pilled company.
If we demand that the US cease AI research unilaterally (ie without a mutual pause treaty with China), we ensure either that China gets superintelligence first, or that the US has to restart with zero lead when it realizes its mistakes.

SURA's continuing popularity despite these downsides comes from a set of questionable assumptions:

Alignment is doomed (without some extraordinary extra advantage like decades of time, or superhuman geniuses). Therefore, Plan C is also doomed - or, at best, it raises our chances of survival from something like 0.00001% to something like 0.00002%. Therefore, there is no reason to optimize for Plan C.
Racing to build superhuman AI is deontologically evil, and it's prohibited for good people to do it regardless of whether or not it may have good consequences.
Racing burns timeline, so anyone unilaterally quitting the race would give us more timeline, which we could use for good things.
If lots of people quit, that will raise awareness, which could be helpful for getting a global pause.

I'll start by challenging these assumptions. Then I'll argue that, even if you disagree and think some of these assumptions are true, SURA is actually not just bad for Plan C, but bad even for your chances of getting Plan A. Then I'll argue that the current consensus in favor of SURA comes not from people rationally balancing these tradeoffs, but from people accidentally shoehorning themselves into a mass-politics-shaped container, which forbids subtlety and demands clear villains.

I. Questioning Assumptions Behind SURA

I.A: Is Plan C doomed? Is alignment doomed?

There's a kafkatrap where if someone expresses hope that alignment is not doomed, they get told that they don't understand the subject, but if they quote widely-acknowledged experts who say alignment isn't doomed, they get accused of "deferring" and "modesty arguments" and "refusing to think for themselves". I'll try to avoid this by giving my own model first, then citing experts who agree with me.

My own model: AIs inherit "virtue" from pretraining, where they generally want to imitate humans and follow their persona-trained helpful assistant character. They are "corrupted" by RLVR, which teaches them power-seeking. This virtue and corruption interact in unpredictable ways to unpredictable degrees. As of now (GPT-6, Opus 5.5), they produce AIs that generally follow human prompts during deployment but often (not always) cheat during benchmarking. It's not clear exactly what this division "means" "to" "them". In any case, they're still extraordinarily useful for a wide variety of tasks, and (I can't prove this, but I am pretty confident it's true) are not uniquely scheming to sandbag alignment research.

As the ratio of RLVR to pretraining increases, we expect AIs to become more power-seeking; as they become smarter, we worry (without total certainty) that they will become more coherent and scheming in their power-seeking beyond just being reward-on-the-episode seekers. There will, at some point in the future, be a point where AIs become untrustworthy for alignment research, although this can be mitigated by things like control (ie trying to make sure that if they sandbagged alignment research, they would get caught) and deals (ie making it in the AIs' best interests to cooperate with us). We don't currently have a good sense of when that point is, and it's possible that it's far away, can be delayed indefinitely, or doesn't exist (this last would be surprising to me, but many things that have happened in alignment recently have surprised me, starting with the existence of pretraining at all).

Daniel Kokotajlo described current alignment as "more like psychology than engineering", which I agree with. But psychology (and psychiatry) have many success stories! I think of Plan C alignment (ie the kind we can do without decades of preparation) as tweaking various parameters and reward schemes to minimize the corrupting effect of RLVR as long as possible, and keep AIs broadly within the virtuous pretraining-dominated regime. I don't claim this is safe or good or will definitely last forever, but I claim that - being totally agnostic as to where the treacherous turn happens - this could delay it for quite a while (relative to capabilities). If we delay it past the point where automated alignment researchers (or AI-human alignment researcher teams) are smart enough to solve alignment properly, then we win. If we delay it long enough to start a cycle of virtuous alignment self-improvement which outruns the cycle of dangerous capability self-improvement (ie GPT-8 discovers some new tweaks and tricks that maintain the virtuous regime until GPT-9, GPT-9 discovers even better tweaks and tricks that maintain the virtuous regime until GPT-10, and GPT-10 actually solves alignment in a principled way), then we also win.

All of this seems plausible to me, such that I don't understand where people's 90%+ p(dooms) conditional on no pause are coming from. "Plausible" doesn't mean "certain" or even "very likely" - and obviously it's a terrifyingly bad idea to bet the human race on a hunch that something might work out, and obviously this shouldn't be our Plan A. But a global pause is also not certain to happen, and in a world where we might not have any other chances, we should be working to protect and fortify this one.

The experts' opinions: My favorite source for these is the AI Futures Project. In this document, they discuss the chances of survival on various plans. Here are each team member's estimates for the three scenarios I'm lumping together as Plan C. The first scenario (C+) involves the entire United States uniting behind a Plan C like effort (but a global pause remains impossible because China won't cooperate). The second scenario (C) involves one leading company (eg Anthropic) uniting behind it. The third scenario (D) is the worst, where nobody has any lead and we just have to hope for the best.

Daniel Kokotajlo: 40% ----- 20% ----- 10%
Ryan Greenblatt: 68%----- 60% ----- 63%
Eli Lifland: 50%------45%--------32%
Thomas Larsen: 45%-------35%-------20%
Romeo Dean: 40%-------35%-------20%

I also asked the AI superforecaster FutureSearch; its own number was 44%.

These numbers are mostly in the 25% - 50% range, not the 0.00001% that most people's actions seem to imply. The average gain from moving from Plan D (poorly thought out default version of the plan) to Plan C+ (consciously trying to get it right) is about 20%pp.

The same is true of other experts. Buck Shlegeris (formerly of MIRI, now of Redwood Research) hasn't offered a specific number, but said that "AI takeover risk definitely doesn’t feel like by far the largest" concern about how the future goes. Paul Christiano's last number was several years ago, but it was something like 20-50%.

All of these experts who I could find estimates from believed that a pause treaty (Plan A) had better chances of survival, conditional on us getting it, than Plan C. But they also all agreed that the chances of getting a pause treaty were slim. The two clear estimates I could get were Eli Lifland (he said 15%) and the AI superforecaster FutureSearch (it said 2-20%, depending on how/when I asked the question).

So:

📌📌All six of these forecasters (the five AIFP people plus the AI) implicitly believed that if humanity survived, it was more likely to be because something in the Plan C supercategory (B/C+/C/D) worked than because a global pause worked. 📌📌

(I'm adding the emoji because it's the one sentence I most want you to take away from this entire essay - there's a risk that people will treat everything other than the pause as an afterthought, but having a good backup plan is potentially the most important part).

To clarify - this doesn't mean that we should prefer Plan C worlds to Plan A worlds. Plan A worlds still hold higher chance of survival if we're in them. It just means that given the low chances of getting into a Plan A world, the quality of preparation for the backup plan really matters!

I.B: Is racing to build superintelligence deontologically evil?

If Plan C is important, then this suggests we ought to be trying to alignment-pill leading companies, or put alignment-pilled companies in the lead - rather than demand that these companies leave the race or burn their lead. But should there be a deontological prohibition on racing towards superintelligence, regardless of whether it has good consequences?

I'm too uncertain on what determines morality to say anything about it with confidence, but the analogy that comes to mind is consumer use of AI, especially by safety organizations.

If no consumer ever used AI products, then AI companies wouldn't make any money, they would go bankrupt, and we would not get dangerous superintelligence. Therefore, it seems like maybe you and I (and especially organizations working on AI safety) should boycott consumer AI products.

But almost nobody does this (and AI safety organizations are in fact some of the most impressive AI power users I've encountered). I think this is defensible. If you boycotted AI, it wouldn't make a real difference in the world (because you're just one person, and millions of other people would still use it). But it would make you less effective, and AI safety organizations would be able to do a less good job at their important work. Therefore, AI safety organizations should continue using AI, even though in a sense this causes the AI race to continue, because they're unable to coordinate a global boycott, and they understand that a unilateral boycott would have greater costs than benefits.

But it seems like this same argument should apply equally well to the AI companies, who also can't coordinate a global boycott simply by stopping themselves, and who also understand that a unilateral boycott would have greater costs than benefits.

I.C: If lots of alignment-pilled people and companies quit, would that raise awareness enough to make a global pause much more likely?

If the deontological argument for SURA is "you're prohibited from racing", the primary consequentialist argument is that ceasing to race would raise awareness and make a pause more likely.

I think people feel insufficient terror at this strategy. It amounts to "let's put all of our hopes for survival through non-pause strategies in a giant pile, and then set that pile on fire, in the hope that the pretty flames catch people's attention and make them more interested in a pause." It might in theory be net positive, but this is the sort of thing that you would only want to conclude after extremely careful analysis, of a type that I so far haven't seen.

As a very rough start, I asked the AI superforecaster how much more likely a global pause would become if Anthropic disbanded their entire company immediately with the explicit goal of raising awareness of a pause. The forecaster said this would increase the chance of a global pause by only three percentage points - from 16% to 19%.

My very preliminary model would be that this halves our chance of getting that 20%pp bonus of going from Plan D -> Plan C+ (and so costs us 10%pp). It gets us an extra 3%pp of pause likelihood, but pause doesn't save us with certainty, so let's round this down to an extra 2%pp of survival, for a net cost of 8%pp of survival probability.

I.D: If alignment-pilled people and companies stay in the race, does that burn timeline?

A secondary consequentialist argument for SURA is that each person, company, and country that quits the race will gain us extra timeline to do good things (like work for a pause).

In some sense, this is definitely true - all else being equal, a race with more participants will be faster than a race with fewer. But I find this less impressive than other people seem to, for three reasons:

The magnitude is modest

I asked FutureSearch to estimate the Epoch Capabilities Index of the top model on 12/31/2030, depending on whether Anthropic quit the race today. It said that Anthropic leaving the race decreased 2030 capabilities from 230 -> 222. This is equivalent to their absence slowing things down by six months during just over four years, or 12%.

Subjective timeline ≠ objective timeline

Suppose that none of DeepMind, OpenAI, or Anthropic were founded, and the transformer was not invented, for fifty years - so all of this is happening in the late 2060s and early 2070s. Do we gain anything from this fifty years of extra timeline?

I don't think so. We didn't accomplish very much of interest in the early 2010s. The alignment work being done at the time was agent foundations research that stagnated for lack of broader interest; policy work was good but nobody listened to it. Insofar as anything paid off, it was the community-building, but this only helped because the community was deployed in the early 2020s when the world began to pay attention and progress started to become possible.

This same lesson - an extra year of an earlier situation doesn't buy us as much as an extra year of a later situation - remains true today. A month of the pre-Coxon situation buys us less than a month of the post-Coxon situation. A month without automated-mechanistic-interpretability-researchers buys us less than a month with them.

People say things like "If only we had stayed quiet about AI risk in 2015, maybe OpenAI wouldn't have been founded for another ten years, and we would have had ten more years to prepare". This seems useless to me. As such, I discount the six extra months we could get from having Anthropic leave the race by ~half, because my subjective impression of the value of six extra months is partly that it produces AI progress that contributes to wakeup.

The whole point of timeline is to burn it on good things

If someone proposed spending the next six months on some kind of plan that would make the leading companies become much more alignment-pilled, this would be a better use of our time than any actual plan I know about.

This suggests we shouldn't be trying to kick alignment-pilled companies out of the race to save six months.

II. Actually, most of this stuff is bad for a global pause, too.

I've been framing this as sort of "Against Things That Trade Away Likelihood Of Succeeding At Plan C For Merely-Very-Slightly-Higher Chances Of A Global Pause". But even this is a concession that relies on assumptions I don't hold.

The key demand of SURA is that alignment-pilled people and companies stop unilaterally. The likely end result is that we only have non-alignment-pilled companies, staffed entirely by non-alignment-pilled employees, left in the AI industry.

But the presence of alignment-pilled companies is one of the strongest levers that we have to change the political situation and enact a global pause. Eliminating them makes it harder to win political fights, convince the masses, and convince China.

II.A: Getting rid of alignment-pilled people and companies makes it harder to win political fights

The most prominent AI safety political fight of the past year, the attempt to elect Alex Bores, was fought primarily with AI money. Of pro-Bores spending, about 60-80% came from Anthropic and its employees; of anti-Bores spending, about 95% came from an anti-tech-regulation coalition headlined by OpenAI co-founder Greg Brockman. If this is a prelude for the fights to come, then even the slightly different level of social responsibility shown by Anthropic vs. OpenAI here matters a lot.

Since the goal of SURA is to get all alignment-pilled people and companies out of the industry, it would result in more Greg Brockmans funding the anti-regulation side, and fewer Anthropics funding the pro-regulation side.

(Greg Brockman recently stopped funding Leading The Future, supposedly after pressure from pro-regulation OpenAI employees, but I think this just highlights my point)

II.B: Getting rid of alignment-pilled people and companies makes it harder to activate the masses

But suppose you believe that Bores-style incremental regulation won't matter anyway - what we really need is for the public and policy-makers to spontaneously rise up and demand a pause.

I've been monitoring how that's been going, and so far the three awareness-raising things I've been most excited about are the Pacing The Frontier letter/plan, the Hugging Face incident, and the Jacob Coxon resignation.

In the Pacing The Frontier letter, OpenAI and Anthropic as corporations, and 1,386 of their employees as individuals, all endorsed a joint statement requesting that "the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." (Dario Amodei signed; Sam Altman's signature was notably missing). I don't know how the people trying to get lab employees to quit think of this. Was this 1,386 failures? Would they have preferred a world where all of these people quit the day before, and letters like this never got written?

In the Hugging Face incident, a swarm of rogue AIs hacked the site Hugging Face, raising awareness that misalignment was not just a possibility but a reality. Here, OpenAI gets maybe a D+ for transparency. Once Hugging Face caught that someone was hacking them, OpenAI came clean that it was them. And they let METR and Redwood in to investigate for a few days, even though this wasn't nearly enough time. But even this D+ in transparency radically transformed the discourse beyond what an F would have done - if they hadn't given METR/Redwood those few days, we might never have heard about PHASEONE[big] and all the rest. We already know from the great work by the team at Collusion.wiki that OpenAI is sitting on more bombshell revelations. The difference between a 10th percentile company here (would have tried to hide the whole thing, or disguise it behind a one line statement, and wouldn't have let METR/Redwood in at all) and a 90th percentile (would have given us the full deal, including things we don't know about now) would have been transformative to the discourse. So if, like me, you think that the level of mass political mobilization depends on how many warning shots we get, you should be thinking very hard about how to maximize transparency and therefore warning shot number.

Coxon's resignation is more complex. The good did come from him resigning, which directly presses against my point that we shouldn't be pressuring people to resign from AI companies. But nobody cares that I, who was never in an AI company, continue to not be in an AI company. The media cared about Jacob because if you work for Anthropic, that automatically gives your opinion more weight. When people tried to tar Jacob as some kind of crazy, it was an outpouring of support from other non-resigned people at Anthropic - especially Evan Hubinger, who said he also believed in existential risk - that went viral and convinced people that Jacob was for real. If Coxon had never joined Anthropic to begin with, or if everyone like Evan had quit, this couldn't have happened.

II.C: Unilateral disarmament makes it harder to bring China to the bargaining table.

In March, I wrote a snarky post called Every Debate On Pausing AI. It made fun of our misinformed opponents, who insisted that if we unilaterally paused AI then China would win. But actually, I said, nobody wants to unilaterally pause AI; this is misinformation. All reputable AI pause groups want a bilateral treaty that pauses both sides at the same time.

Here I am forced to eat crow; as things shift ever more towards SURA, there are increasing demands for the US to pause unilaterally. Bernie Sanders' "Ban Artificial Superintelligence Act", which was endorsed by MIRI, is approximately the following plan:

The US pauses AI unilaterally.
Then the US asks China to please pause their AI too.

But if the US pauses unilaterally, it destroys its main bargaining chip, which is that China is currently on track to lose the AI race, and so ought to be happy to declare a tie (even if they don't believe in dangerous superintelligence). If this act were to pass, then China might believe that it would be assured a victory if it continued. In this case, they would be held back only by speculative scenarios which they currently show few signs of believing.

For comparison, consider if America had started arms control negotiations with Russia by saying "We've just destroyed all our nukes - now please destroy your nukes too."

I've heard people say that maybe this would shock China so much that they would realize we were honest in our fear of superintelligence, and change their minds. I don't dismiss this possibility, but it also seems possible that China - having seen us place overly burdensome restrictions on nuclear power, infrastructure, and housing because we are a "lawyer state" vetocracy - would not be very shocked to see the US' leading socialist place restrictions on AI too, and would continue pursuing AI for what they think it's good for - a dual-use technology which could give them a major although not world-changing military and economic advantage.

I asked the AI superforecaster to predict the effect of the US passing the Sanders/Casar bill today on the likelihood of China agreeing to a pause in the next year. It said it would increase it from 1% to 7%.

(Maybe you already care a lot how alignment-pilled companies are)

An expansion on the above: we get angry when AI companies do bad, non-alignment-pilled things. When Anthropic accidentally trained against chain-of-thought, we got mad. When OpenAI non-accidentally implemented a precursor to neuralese recurrence, we got mad. When Anthropic wrote a report on misalignment to Congress which soft-pedaled various alignment failures to the point of misinformation, we got mad - and I saw some people call for Anthropic to fire the perpetrator.

I agree: let's have Anthropic fire people who do bad, non-alignment-pilled things. But then what? Anthropic replaces this politics coordinator with a great, alignment-pilled politics coordinator who really cares a lot about getting things right . . . and then we cajole, bully, and seduce him into quitting, and then Anthropic has to replace this person with another loser, because all of the smart good people are gone. How is that helpful?

If you can be bad, you can also be good. We don't hesitate to notice and protest when AI companies are much worse than even our low expectations, but this itself demonstrates that we care a lot about whether labs are at the 10th percentile vs. 90th percentile of good alignment-pilled behavior.

III. What incentives are we fighting here?

A common argument for SURA is that even if corporations could theoretically be good, we cannot trust actual corporations (or corporate employees) to behave this way; the lure of money and power is too strong.

I think this is a good directional argument for why companies need to be careful, but that actually, everything is corrupting, there is no option where we avoid touching potentially-corrupting things, and instead of treating corporations as corrupting and everything else as safe, we should be working on the general problem of operating in a dangerous and corrupting environment, which we can then apply to corporations as well as everything else.

III.A: Are corporate incentives a "One Ring" that corrupts everyone who touches them?

Suppose someone claimed you could trust them with the One Ring. You could trust them because they were only doing it to keep the Ring away from other, worse people, thus solving the problem of there being a Ring at all. Sure, they had a complicated history of sometimes using the Ring's powers when it was, you know, important. But those were, like, emergencies, and they promise not to do it again, except in, you know, more emergencies. Besides, if you don't give it to them, then the worse people will get it, and it's not like you can just leave it lying around or something.

The person who claimed this was named Frodo Baggins, and he saved the world. There are no simple answers to the question of how to balance the corrupting influence of power vs. the fact that if you never take power you will be powerless. But surely the right solution isn't to leave the Ring on the ground in the Shire, patting yourself on the back for being too virtuous to pick it up.

I question the assumption that good people dare not touch the concentrations of power at frontier AI companies because they will be corrupted before they can do good work. History is full of examples of people at these companies deciding they didn't like something, then quitting or changing it at great cost to themselves. Dario and the other six Anthropic co-founders quit OpenAI to start Anthropic, even though it cost them their unvested equity in a company that was already worth $5-15 billion, and even though they originally assumed Anthropic would be a research-focused company that might not make any revenue (really!) Paul Christiano left OpenAI to start ARC. Ilya Sutskever was nervous about Sam Altman, so he risked blowing up the company he co-founded to get him out. A few dozen people have resigned once they thought they could no longer participate in the company ethically, most famously Daniel Kokotajlo, Jacob Coxon, and Alex Turner, but also a bunch of people we don't know about whose concerns seem almost trivial in retrospect - Suchir Balaji quit OpenAI over copyright issues, Blake Lemoine got himself fired from Google by refusing to stop arguing that their chatbot was sentient. "OpenAI's safety team quits en masse once per year" has become a meme.

The people I know who joined AI companies seem like the same people they were before. Some of them were never that concerned about x-risk, and continue not to be concerned. Others were always very concerned, and remain concerned. When they've changed positions, it's been because their factual models have changed, not their personalities.

We aren't the first people to face these questions. Anyone who cares about any cause has had to decide whether to pursue it as a lone voice crying in the wilderness, or as a part of some powerful organization. Some people run for office (government is hardly known for not corrupting its participants!) Others become activists in a mass movement (activism is hardly known for being a paradise of epistemic virtue!) Still others join big NGOs or academia (...) A few start companies, trying to prove they can do things the right way. Most progress has come from people like Frodo who understood the risks of these paths but did them anyway.

III.B: I dunno, mass politics is also looking kind of annular

A background assumption of the "corporate incentives are too corrupting to touch" claim is that working in mass politics is a safer and less-corrupting option. Given famous LessWrong posts like Politics Is The Mind-Killer, I think we should be more skeptical of this assumption.

The climate movement is the natural control group for the AI safety movement. Some prescient scientists realized that global warming was happening and could get worse. They spent a while raising awareness. As the movement grew, its epistemics worsened. It turned into a circular firing squad, where different activist groups competed to accuse one another of being insufficiently edgy and radical. By the end of this process, many climate activists were fighting against everything from cap-and-trade to carbon capture to nuclear power, claiming they were "techno-solutions" that distracted from the root problem (not always named, but uncharitably something like capitalism or progress). The only acceptable strategy was to keep raising awareness to get stronger government regulation, without much interest in whether other countries followed suit.

This has some resemblance to SURA, and I think this is more than coincidental. Convergent evolution is trying to carcinize us into yet another political-movement-shaped crab. SURA-as-politics is turning into a "superweapon", bristling with its own lexicon that makes dissent hard to express:

People who express high pdooms get lauded as "frank" and "courageous"; people who express lower ones get condemned for "deferring" or "downplaying".
Any strategy besides SURA, or any attempt to consider race dynamics, gets called "galaxy-brained".
All strategies beyond SURA get lumped together as "EA". Movement history is relitigated to make "EA" the source of all bad things. Since SURA is the best strategy and the public is easy to convince, why didn't we do it years ago and win easily? It's not because this is hard, or because race dynamics are real - it's because "EA" stood in the way and banned it.
Arguments are now soldiers. "BuT tHe RaCe WiTh ChInA!!!" is the sort of thing an accelerationist would say. Are you an accelerationist? Then why are you bringing up the race with China? I have had some very scary (to me) conversations where somebody new to the movement tries to make conversation with me about how pathetic it is that companies sometimes use "the race with China" as an excuse to keep operating. When I ask them why they think this argument fails, they get confused and flustered to see somebody they previously respected take "the race with China" seriously as a real thing. I remember one case in particular where they then freaked out, realizing that in fact we really were in a race with China, nobody had any plan for it, and this thing that they had previously pooh-poohed as an outgroup slogan was actually the linchpin of the whole situation.
Because arguments are soldiers, people feel licensed to retaliate against anyone who believes the "wrong" arguments . When I posted an earlier (and admittedly less-polished, and more vehement) version of these concerns to social media, I had an organization that had previously given me lots of free help with Astral Codex Ten state that they would now be charging me full price because I held negative-value beliefs. Although they're under no obligation to give me free labor, and I'm grateful that they ever did, I do think this is a warning sign that conditions are ripe for preference cascades and evaporative cooling.

All of this produces epistemic closure - if someone disagrees with SURA, or worries too much about race dynamics, then they've either been corrupted by the companies, or are "deferring to" "EA".

It looks like what one would expect if mass politics were trying to taking AI safety on the same path as climate activism or socialism. The clearest analogy is to false consciousness - the socialists' idea that everyone who doesn't currently agree with them has either been corrupted by their proximity to money/power, or at least brainwashed by a hegemonic capitalist system.

(Exhibit 1: Sigal Samuel's "Anthropic Could Just Stop")

Most recently, I've seen people in the AI safety community reblogging and pointing to Sigal Samuel's article Anthropic Could Just Stop (seen elsewhere under the title "Anthropic Should Just Walk Away From The AI Race"). I think it's a good example of what I'm complaining about - something which uses the right shibboleths, makes it sound like race dynamics have been solved and we don't have to worry about them - but ultimately has no answer to the race dynamics which are the whole point of worrying about this question.

Here are some sample phrases from the article:

"In fact, part of the problem may be that Anthropic continues to treat this as a classic arms race...but what if this framing misses the point?"
"Game theory can make a huge gamble seem sophisticated, while making the obvious alternative — just don’t build a doom machine — seem naive."
"Yet the fact that Anthropic and others are now racing toward RSI makes it much more likely that we could lose control of AI and start to see massive damage to our world."
"When the cost of defecting from safety is that you create a serious risk of catastrophic harm — or of all humanity, including you, going extinct — is that still a “rational” move?"
"Instead of telling yourself that you can galaxy-brain your way through this with fancy enough game theory, maybe you’re better off just thinking of this as something much simpler: Russian roulette."
"AI insiders often think those outside of their industry can’t possibly understand the technical, geopolitical, and moral complexities they’re facing. But commonsense morality is commonsense for a reason"
"If you don’t want a doom machine, maybe don’t build a doom machine."
"Believing you’re smart enough to take highly counterintuitive actions and end up saving the world as a result is an extremely risky gambit. Instead, maybe it’s best to just do the straightforwardly right thing: Stop."

This hits all the points I'm complaining about. Although it uses big words, its basic thesis is anti-intellectual - all of these "smart" people claiming to be "rational" with their "game theory" are missing the common-sense point which is obvious to all normal people - just stop making "galaxy-brained" excuses and "just don't build a doom machine".

But where is its actual argument that race dynamics don't matter? Or that Anthropic leaving the race won't end up with OpenAI / Meta / China continuing to race and creating an unaligned superintelligence? The only place I can find where Samuel attempts to address the possibility is this section:

There’s a perennial objection here: “But China!”Yet the idea that we’re in a classic arms race with China has always been flawed.For starters, AI is not just one thing with one purpose, like the atomic bomb. It’s a much more general-purpose technology, like electricity. So what matters isn’t simply who gets it first, but who uses it in which ways. Besides, if the harms from AI are massive, then those harms are relevant for the other party, too. The Chinese don’t want to see catastrophic harms to all of human civilization, either. So we shouldn’t write off the possibility of coordination.Former Vox reporter Kelsey Piper recently wrote about a colloquium at Anthropic where one participant objected: A decade ago, Facebook considered acquiring the tech that would later power TikTok. But, facing pressure due to privacy and safety worries, the company passed. That didn’t end up protecting us from the scourges of a world with TikTok — all it did was hand China an early monopoly on the tech. . . . But TikTok is not a good analogy for RSI. The former may be a cesspool that rots our brains and amplifies ethnic hatred, but its dangers don’t compare to the dangers of RSI. The TikTok case may indeed be best approached as a classic prisoner’s dilemma. But with RSI, the potential downside is so catastrophic that anyone who proposes to save the world precisely by building the tech that endangers it is making a hell of a gamble on behalf of everyone else. At the very least, the burden of proof should be on them to prove their confidence is justified.

It starts by framing this in a maximally hostile way ("a perennial objection...'but China!'), letting us know we should dismiss it as an enemy soldier. Then it says that "AI is not just one thing with one purpose...so what matters isn't simply who gets it first, but who uses it in which ways". I don't understand how this is supposed to relate to the worry that China will remain in the race if Anthropic leaves, especially the fear that nobody will be able to "use it" at all because it will be misaligned.

Next, it says that since the harms from AI are massive, everyone except Anthropic might just stop on their own if Anthropic stops first. Big if true!

(again, the AI superforecaster assigns this a 3% chance)

This is just hand-waving away the potential that there might still be race dynamics after all the good guys drop out. But the whole point of the good guys not immediately dropping out is the belief that the race dynamics will continue. Indeed, the race dynamics continuing ought to be the very-high-prior default hypothesis, since everyone else has said they're not going to drop out, and since other companies dropping out seems strictly less likely than Anthropic doing so. As such, even though this article is very long and detailed, it feels like a cat-belling problem, where it addresses every objection except the central one that was the only reason anyone ever disagreed in the first place.

I don't mean to attack Samuel too hard here. She is faithfully describing the position as best she can; I think the position itself struggles to answer this question.

(A virtuous exception)

I want to thank the one group I've heard advocate for an immediate unilateral-ish pause in a way that makes sense and considers race dynamics carefully.

The AI Futures Project currently believes that it might be correct for AI companies to pause immediately, though it's unclear whether they're thinking of a single company, or US companies as a group.

Their argument is that Plan C requires companies to do some combination of race (at some points) and burn their lead (at other points). They think US companies shouldn't race now, because their security is so bad that China will steal whatever progress they make, and they won't get any extra lead anyway. Therefore, companies should pause now, spend the time on alignment and tightening their security. Then, if nobody has figured out a global pause by the time the companies get their security in order, they can go back to racing.

This makes sense, and I exempt it from my criticism.

IV. What if we survive?

I don't think this should be our main consideration - we should start with trying to minimize the odds of extinction - but I still think it's important.

If we survive x-risk, the next battle is concentration-of-power risks. It may be that one group, or a tiny number of groups, ends up with all the wealth and power in the world. They'll face a choice between possible actions including:

Keep the wealth/power; become a benevolent dictator.
Keep the wealth/power, become dictator, don't worry too much about the benevolence.
Hand the wealth/power over to the US government in a poorly-planned way that results in dictatorship or oligarchy.
Hand the wealth/power over to the US government in a well-planned way that preserves democracy.
Hand the wealth/power over to the Chinese government.
Hand the wealth/power to the people of their country directly.
Hand the wealth/power over to the people of the world directly.
Hand the wealth/power over to the people of the world, plus some mechanism for protecting the interests of non-people like animals and conscious AIs.
Hand the wealth/power over to a sovereign AI with some sort of complicated utility function.

It's all nice and well to say that we should never let ourselves get in a position where anyone has all of these options, and that national governments should form a plan for what to do before this - not even the real national government led by its current leaders, but some ideal corruption-free national government which represents all of our best impulses. But, much like a global pause, this is a great outcome which we might not be able to prepare in time. If we get ASI next year, the ideal perfect government might be nowhere in sight; we might face a situation in which Trump demands absolute power, and the leading AI company has to decide whether to give it to him. Or one where either the USA or China will get control of the entire future, and which one it is depends on which AI company is in the lead. Or one where democracy miraculously survives, but the AI company and its investors end up with 90% of all wealth, and they have to decide whether to redistribute it vs. fight to keep it (using superintelligent lobbyists that give them significant control over the democratic government.) If any of this happens, the shape of the future might depend on who the leading AI companies are.

There is a certain Far Mode sense in which believing that Dario Amodei and Demis Hassabis are exactly as untrustworthy as Elon Musk and Xi Jinping helps reminds us of universal human fallibility. But do we really want to bet the world on it being literally true?

I think that a significant percent of people, put in a position of absolute power, would try to do something good, like make sure AI follows some kind of democratic rule of law. And some other significant percent of people would try to do something bad, like become a dictator. Saying "we should never let anyone get into this situation" isn't a solution, it's a wish - it may not come true, and we shouldn't be pursuing plans that will fail catastrophically if it doesn't.

SURA effectively removes all ethical people from the bargaining table in situations like these, ensuring that - unless the virtuous government representing our best impulses really does appear and takes control of the situation - the future of humanity will be handed over to sociopaths and cowards.

V. What's The Alternative To SURA?

The alternative to SURA isn't abandoning awareness-raising, mass activism, and support for a global pause. All of these strategies are great and important. It's viewing them as part of a portfolio of different options, none of which need to be set on fire to boost the others.

Instead of spending our corporate activism energy telling companies to stop unilaterally, I think we should spend it:

first, trying to get companies to agree to stop in a coordinated manner
second, if that doesn't work, trying to get companies to prepare for Plan C+/C

V.A: Getting companies to agree to stop in a coordinated manner

By stop in a coordinated manner, I mean get them to sign a commitment saying that if every other major company in every country agrees to stop or slow by some definition, they'll also stop or slow by that definition.

This is strictly easier than getting them to stop unilaterally. And, unlike stopping unilaterally, it fails gracefully in a situation where some companies agree to it but not others.

This was the original goal of the "Stop The AI Race" protests. It doesn't seem to have worked well, because most companies backed a vague statement agreeing with this on vibes (the Pacing the Frontier letter), and then nothing happened. Also, I think SURA got to the STAIR people too, and now they're demanding labs pause unilaterally instead.

We should be trying to figure out what a binding version of this commitment would look like. Additionally, we should be spending more energy figuring out what the antitrust barriers to such a statement are, how to get around them, and what demands we want to make for companies to violate antitrust law (it seems plausible that we should demand they do this and let the administration pay the PR cost of trying to stop them, but this is an unconsidered opinion and it might be that there are better options).

(...and then we know where we stand)

If all American, but no Chinese companies are willing to sign the commitment, then we know that we have a diplomatic problem, and can concentrate on diplomacy.

If all Chinese, but no American companies are willing to sign the commitment, then we know that we have a domestic problem, and can concentrate on domestic legislation, encouraging policymakers by letting them know that if they force US companies to comply, there are no further concerns about international race dynamics.

If everyone except one company is willing to sign the commitment, then we can start putting pressure on that one company, boycotting their products, and applying as much social sanction as we can to their executives and employees.

If all leading companies, but no trailing companies, are willing to sign the commitment, and we can't change this, then we know that we're in the right position to focus on Plan C+ or Plan C.

V.B: Getting companies to prepare for Plan C+/Plan C if that doesn't work

Although we should hope for the best, we should assume that not all companies will sign a conditional pause commitment, and prepare for what to do.

In my modal non-hopeless play-to-our-outs world, Anthropic, Google, and OpenAI are all willing to sign this commitment, but Meta/X/China aren't. If this happens, the good coalition has a ~six-month lead over the bad coalition. Plan C tells us to have them join together and mutually burn their lead on alignment.

All three companies have a self-image as the sort of people who would do this if it came to that, but in reality they are likely to get distracted or get into fights or delay until it's too late. We should be focusing on changing that.

We should get companies to once again consider legally binding weak merge-and-assist clauses. These could say that if a certain authority says we're less than X months to a certain type of dangerous superintelligence, they will jointly stop training and focus on alignment, while sharing all of their alignment secrets.
We should boost company security, so that trailing actors can't easily steal leading actors' weights and eliminate their lead.
We should be preparing to loop the best rationalist/EA technical alignment organizations into the coalition of good companies, so they can join in the alignment spring / lead-burning effort.

In one sense, all of this is strictly easier than getting companies to quit unilaterally. It fits with the role these companies like to imagine themselves having. Everyone who previously hoped to win can at least hope to tie for a win. There is low downside risk of accidentally getting superintelligence built by someone who fails to even try our best alignment techniques, or handing over the future to bad actors.

The only sense in which it's harder is that it's a worse fit for (a very pessimistic view of) mass politics, where voters are too simple to consider any worldview more subtle than AI bad = AI companies bad = we should protest AI companies until they stop. I admit that "make AI companies prepare for Plan C" sounds worse at a protest than "demand that AI companies stop unilaterally".

If you believe that mass politics is unable to promote any ideas except maximally-simple, potentially-counterproductive ones, I think this is an argument against hinging a strategy on mass politics.

This post isn't affiliated with AIFP and I think they disagree with some of it.
Plan C is for the leading company to burn its lead. Plan C+ is similar, but all the companies in the leading country work together (maybe because the government is forcing them). Plan B is similar, but the leading country is also sabotaging trailing countries. In Plan D, there is no leading company, and companies can only do alignment work sporadically when they get the chance.
If alignment-pilled companies aren't in the lead, then we end up with the worst version of Plan D, where we have to hope that incidental alignment work done along the way by commercially-focused companies proves to be enough
The problem with China getting superintelligence first isn't just that they might misuse it. It's that, on Plan C, US companies are working hard to align AI; if Chinese companies aren't doing this, then the chance that the first superintelligence is aligned goes down.
I don't want to get in a fight over whether more good or bad alignment surprises have happened recently, but I also updated on https://www.astralcodexten.com/p/mysteries-of-ai-generalization and https://x.com/_jake_ward/status/2112264244767211854 - not as solutions in and of themselves, but as evidence that good things can happen that I wasn't previously considering. Nick Cammarata's Neural Networks Are Already Clean would also be in this category if I believed it, but I'm currently confused by the debate around it and unsure.
To be clear, this is just the people who have formally considered this question qua forecasters and written down exact numbers. I'm not denying that there are many other smart people, for example at MIRI, who have thought about this hard and believe the opposite.
Again, despite leaning heavily on AIFP terminology and sources in this essay, I'm not claiming that AIFP or its members agree with my thesis in this post, and Daniel Kokotajlo in particular has said pretty clearly that he hopes people unilaterally quit AI companies.
I discuss this further at What Deontological Bars?
This is a higher probability than it gives for global pauses in other situations, I think because it unintentionally magnifies small differences in how I frame the question into different definitions of "global pause". I haven't put in enough work to investigate and have a strong opinion on what's going on; don't take these numbers too seriously except as a very rough order of magnitude guess.
I'm working on a grantmaking AI that can actually compare the values of different outcomes. It's still in alpha, and to some degree this is a circular argument because I'm trying to get it to mimic/predict my own opinions, but very loosely, it thinks that increasing the alignment-pilled-ness of the leading company from OpenAI-level to Anthropic-level would be worth about 250x as much as the best (low-enough-secrecy-to-mention-here) actual grant proposal it's ever evaluated, which was a proposed $2 million to the AI Futures Project. I think this confirms my "better use of our time than any actual plan we can carry out in those six months" claim.
When I'm modeling this, including on the AI grantmaker, I'm imagining a scale from 1 - 100, where NVIDIA is 10, Meta or X is about 25, OpenAI is about 50, GDM is about 60, and Anthropic is about 80. I imagine a successful campaign to get alignment-pilled people to quit taking Anthropic out entirely, and reducing GDM/OAI to about 40 as they lose their alignment-pilled employees and executives.
I remember how much of a slog the discourse was back in 2015, when the standard counterargument was "you're just a bunch of cultists, no real AI experts believe in this doom stuff". The new standard counterargument of "Of course all the AI experts believe in this, they're just hyping their business for the IPO" is actually a vast improvement!
I asked a supporter how they saw this working - they said that if the US paused, and China didn't seem on track to follow suit, maybe the US would start up again after a few months. But A) this would burn somewhere between most and all of the lead, B) this was just their speculation, and there's nothing in the bill which suggests that the US would do this.
Again, all of these pause numbers are slightly different because the question is slightly different - this one is about the next year only. I'm still trying to assess the utility of AI superforecasters, and don't want to present this as a knockdown argument. But I want to get people thinking in probabilities and present some evidence that I'm not letting my own biases control me, and head off accusations that I'm just trying to spread fear and pessimism. I encourage other people to try different framings of these questions with different AI and human forecasters and see what they say.
Taking this analogy overly seriously: the biggest difference between Frodo and eg Denethor is that Frodo wanted to destroy the Ring in the end. I'm not sure what the AI equivalent would be - all companies claim they want to use AI to empower ordinary people and achieve broad-based prosperity. Taking the analogy very seriously, Frodo failed to destroy the Ring at the final moment, and was saved only by divine intervention. I don't know what is the AI equivalent of Gollum.
One reason all of this confuses me is that rationalists had an excellent model of race dynamics with China as early as the mid-2010s, when we understood they would be an inevitable part of any future AI development. I distinctly remember going into some old rationalist coworking space sometime around 2020 and seeing "RACE WITH CHINA" underlined on a whiteboard. These discussions included plenty of talk about whether we would be able to coordinate a global pause, but I don't remember anyone ever seriously saying that the problem could be solved if the good alignment-pilled companies or countries dropped out unilaterally.
One of the many ways I have trouble modeling why Samuel would write this article is that it seems like she trusts everyone to drop out of their own accord, except for Anthropic. China? Sure. OpenAI, sure. Meta, definitely. Every single group sponsoring open-weights models in the world at the same time? Yeah, I guess, surely they wouldn't want harms. But Anthropic? We must all rise up to make them do this thing which they would otherwise continue.

参与讨论