Extreme Power Concentration: A Map and Research Directions

AI is increasingly capable. Suppose we're lucky and the alignment problem gets solved, so AI reliably does what its operators want. But we’re not done yet: it lends immense power to those controlling it by progressively replacing the need for humans, disempowering many. [1] AI could enable extreme power concentration, producing states unaccountable to their citizenry, and more stable and totalitarian than any previous ones have been. While LessWrongers have somewhat discussed this (e.g. in Gradual Disempowerment, Intelligence Curse, AI-enabled coups and the like), I still think there are far too few people working on this! As far as I can tell, it’s largely Forethought (who do macrostrategy), Formation Research and EuroSafeAI (shameless plug), as well as some distributed work here and there. Most people seem to agree it’s a large concern, and that few are working on it, but many aren’t sure what to do. [2] I hope this post deconfuses the space and helps technical people contribute, hopefully increasing tractability. In this post, I want to Introduce the problem (skip if you’re acquainted [3] ), Introduce a useful frame to think about it (the classic IEMP model), and some of the leaf-node threat models there. (e.g. surveillance) Give some potential recommendations for projects and action, inviting discussion. In summary, I’ll explain this graph, and give recommendations. I think this is an important and neglected problem; I'm uncertain about magnitude; this post maps the problem rather than forecasting it. It’s also preliminary, hopefully contributing to discussion on the topic, and omits details (with some relegated to footnotes). Different opinions and additions would be much appreciated! Furthermore, I will generally assume that operator-alignment is more or less solved (a tall ask, I know). Why AI could centralize power Most stories of extreme power concentration are pretty specific or concern one pathway (such as AI-enabled coups). For the sake of this post, I’ll focus on the general case. The two core drivers are fairly simple: AI automation of labour, and runaway AI R&D giving a decisive lead. The first is the main engine: if you don't need humans to do the work, they lose leverage, and you don't need to listen to them. The same goes for holding power itself: historically that required many cooperating humans (soldiers who could refuse, officials who could slow-walk), whereas AI agents could be far more loyal. [4] Put in economic terms, capital becomes a substitute for labour, and because capital is much more concentrated than labour, control over AI labour concentrates in the hands of capital holders. [5] The second driver, a decisive lead from runaway AI R&D, then amplifies this: whoever it is gets more raw economic, military, or cultural power. [6] (Though note the amplifier isn't required; automation alone shifts power even in a multipolar world.) But how should you think about power? It can be a bit confusing to think about, especially since human society is more multi-part than an AI system. Breaking down sources of power, I have grown to like the well-established IEMP model by Michael Mann . The IEMP model divides sources of power into ideological, economic, military and political dimensions. These can be converted into each other (a businessman can support a politician, or Taylor Swift can convert appeal into money), but usually , they tend to be surprisingly independent: A powerful FBI bureaucrat (political power) may struggle to get an absurdly high income or become famous (economic and ideological power). So, how does AI automation of labour concentrate power? A few arguments: For economic power: Hasn’t previous technological change always created new jobs despite worries? Usually true. [7] But previous tech usually complemented humans. It wasn’t trying to surpass human beings in all possible areas the way the big labs are trying. If AI automates parts of labour, the labour share of GDP (currently at ~55%) will start to fall. [8] That means more economic power goes to capital from labour, and capital is much more concentrated than labour. Importantly, this holds regardless of who captures the surplus: competition may push the gains from the labs to hardware makers or downstream businesses, but all of these are held by capital. Who wins the intra-capital fight doesn't change the labour-to-capital shift. [9] For military power: In modern states, you tend to need many soldiers to comply with you to get things done. If drone armies or, more speculatively, robot armies become widespread, this again concentrates power away from soldiers to those who control these systems. For ideological power: Once again AI systems may ultimately outcompete humans in quality and quantity. AI models are already more persuasive than even expert canvassers and persuaders in normal chats according to a UK government study. [10] They are also used in the ideological space a lot: Most obviously, recommender systems have outsized impact on our societies, commanding a dominant share of compute at Meta and Google. More generally, one could imagine AI systems increasingly outcompeting human beings in the attention economy, though I currently suspect demand for the human-premium will win for now. I would get particularly worried if AI-human relationships become normalized. For political power: Political power is largely downstream in this model: economic, military, and ideological power all tend to convert into it, which is why it's the natural endpoint of concentration, but also one of the classic redistributors, and thus the one most worth defending. Economic power often translates into political power. Within a state, Acemoglu and Robinson postulate that democracy is a contract due to bargaining between capitalists and labour. Classic Democratic Conditions theorists Moore, Lipset and Dahl all note the extreme importance of wealth for democracy, with Moore famously postulating ‘No bourgeoisie, no democracy’. It is surely no coincidence that wealthy countries are much more likely to have stable democracies. [11] This means the tax base would also shift away from income taxation, with many states becoming akin to rentier states . These rentier states (most commonly, oil states) usually don’t have a great political track record. Between states, the US, being 26% of global GDP (but only 4.2% of the global population) pushes far above its weight, and not only militarily. Military power often translates into political power. Within a state, it is still somewhat debated how much broad conscription contributed to democratization, but it surely helps somewhat. Between states this is also fairly obvious (although economic/political power may be more important in the current order). LLMs could make coercion much easier at scale, including surveillance or lie detection. More speculatively, you could imagine AI models being used in government for different operations. Either AI representatives, or in the civil service for efficiency or transparency reasons. Those are the main factors, but much was omitted for brevity. A summary can be found in this graph, with some plausible leaf nodes as well. Research Projects Below I have linked to RFPs and other people’s suggestions. What follows is my (opinionated) additional ideas of what works, take it with the usual grains of salt. I suspect policy work will be the biggest lever, but I am generally more technical. If you are coming up with ideas, I recommend checking out previous work, or starting de novo from the IEMP model. Of the four factors, economic power seems hardest to solve for technical people. I don’t think this is an accident: The economy is fastest to adapt based on capability, and less regulated than others. (I suspect, compared to innovation, that policy will make a big difference here: The public response to AI will probably largely be driven by economic effects rather than e.g. catastrophic risks. But note that…