Greater good arguments in AI safety

Back in the before-time of the 2010s, when EA was still known mostly for funding health efforts in the developing world, there were many baskets of criticism floating around from the left. The most prominent was a cluster centered around EAs being white men, and therefore suspicious. Another category, in my experience the second-most common, was that in attempting to rectify the problems in the world, EA was masking, whitewashing, and misdirecting efforts away from solving the "real problem", usually capitalism but often other contradictory forces.

These arguments were often not specific to EA, but mainstream arguments against charitable giving in general. On any Hacker News post about a billionaire giving to charity, one of the first few comments would inevitably be this Oscar Wilde quote:

Just as the worst slave-owners were those who were kind to their slaves, and so prevented the horror of the system being realized by those who suffered from it, and understood by those who contemplated it, so, in the present state of things in England, the people who do most harm are the people who try to do most good.

These arguments have a long history. Anticapitalist critiques of charity go back to Engels and Marx, who called charitable givers a bourgeois faction intent on "redressing social grievances in order to secure the continued existence of bourgeois society." In their most explicit form these critiques had the following shape:

• There is a "root cause" of most societal ills, such as capitalism.
• Billionaires giving to deworming efforts addresses the symptoms of this problem, such as poverty and disease. But it's not treating the root cause.
• Additionally, a side effect of such giving is that it gives people reason to tolerate the current system better, or even begin to like their oppressors, which gets in the way of fomenting change.
• So actually, deworming people in Africa, or giving legal aid to poor people in the U.S., is net-negative. You should just spend all of your time trying to change capitalism.

While you could argue with these proposals on the object level, I think most people have an intuitive sense that this line of thinking is fraught, independent of whatever political program is mentioned. One general concern you might have is that if you're going to claim that the first-order effects of someone's good work are swamped by some second-order effect, particularly some second-order effect on optics, then the burden of proof is pretty high. And clearly, the people making these arguments do not have a good enough model of political economy to clear that bar. Not only is it not obvious that billionaire charity makes people like billionaires more, it is also not clear that the detractor's preferred intervention (namely, jettisoning capitalism) is effective at improving the world.

Recently, many people on LessWrong have been making structurally similar arguments about prosaic AI safety work. Here are two such shortforms with >100 karma presenting this argument, however the thesis has also been presented many other places. In general, the argument goes like this:

• Aligning superintelligence is very difficult.
• Prosaic AI safety measures, such as chain-of-thought monitoring for broken RL environments, address symptoms of misalignment but don't solve the underlying problem, or at least don't scale to superintelligence.
• Additionally, getting better at aligning current intelligences means that there are fewer containment breaches, and so there is less support for the best possible solution, an AI pause.
• So actually, you should not do chain-of-thought monitoring. Ideally you should spend all of your time trying to Pause AI, but if you don't want to do that you should do nothing.

The people who make these arguments make them in good faith. Nevertheless, I have both object-level and meta-level objections to their line of reasoning. My first object-level problem with these arguments is that I do think that prosaic AI safety efforts reduce the chance of runaway misaligned AI - even if they don't scale to superintelligence - and that the people making these arguments have confused theories of change. I've tried to articulate that disagreement in more detail here, but in no particular order:

• We are rapidly approaching AIs that are sufficient for bootstrapping recursive self-improvement. Current AIs inside companies like Anthropic already have a lot of say in making decisions to design their successors, so "mundane alignment of present-day AIs" is affecting the safety of future models.
• AI progress is not currently bottlenecked on alignment. This seems like something that alignment people say copefully but is wrong. The two things that matter for AI progress are frontier labs' data efficiency and compute efficiency, two highly quantifiable metrics that can be improved even with models that don't care a whole lot about people or generalize well to other tasks.
• Many techniques, while not silver bullets, raise the ceiling for the most capable model we can use safely. This delays AI takeover, and therefore allows us to develop new prosaic safety techniques, or refine the ones we have. Techniques need not "scale to superintelligence" in order to be helpful in this way.
• Arguments that current alignment techniques break down at some level of intelligence do not usually allow us to predict when they should break down, or at what level of capability. And if we can't tell when they break down, they might very well break down after AI models are then capable of making further prosaic contributions, or else make fundamental advances in e.g. mechanistic interpretability.
• Further, the entire paradigm of asking whether a particular technique "scales to superintelligence" seems wrong and confused to me, somewhat akin to asking if hashing passwords "scales to APTs". The question to ask is not whether a technique is sufficient to contain superintelligence, but whether it could help guard against correlated failures in other components. Derisively call this a "grab bag" approach if you want, but that phrase can be equally used to describe many useful practices in fields like information security.

My meta-level objection is that "AI safety of superintelligence" is a pre-paradigmatic field for which there are no feedback loops, and indeed no superintelligence, and so sacrifices to the greater good are even more speculative than when they're made in service of political ideologies. That doesn't mean that every approach is equal, and it certainly doesn't mean that attention is correctly allocated. But it does mean we should be cautious about encouraging others to abandon approaches that work on current models, if the counterfactual is for them to do nothing.

1. I am lampooning a bit, but not by much.
2. From "The Soul of Man Under Socialism". When I started writing this post I was under the mistaken impression that this quote was from Frederick Douglass. In My Bondage and My Freedom and other places, Douglass actually repeatedly claimed the opposite. I find it telling that the actual former slave who I thought wrote this quote believed that good treatment of slaves encouraged them to meditate on their own situation, and that the quote was instead written by a white socialist several decades after slavery had been abolished in the U.S. and Britain.
3. As an unstated corollary, the evil capitalists, such as the guys involved in the BP Oil Spill disaster, are the True Heroes of the revolution, inasmuch as they do not then use their wealth to cement political conditions.
4. As an unstated corollary, Elon Musk is currently the most effective altruist, because his model is very likely to breach containment in the near future, and he is not currently reducing time-to-AGI except indirectly through competitive pressure, because Grok is not at the frontier.