US data centres ‘are short six NYCs of electricity’

Not everything that Elon Musk says is stupid. Sometimes it’s just obvious. Here he is, speaking about AI data centres at the G20 Innovation Ministerial Summit in North Carolina a couple of weeks ago:

There’s quite a crisis of power. [ . . . ] The consensus at this point is that there will be a significant power shortfall next year. So, not like distant future. I believe the consensus estimate among analysts who follow the AI space very closely is that there will be at least a 15-gigawatt shortfall of power in 2027 for AI chips. The rate at which AI chips [are] being produced has been rising incredibly rapidly. They’re rising on the order of 40 to 50 per cent a year. But power available outside of China has been rising at 10 per cent to 20 per cent a year. Obviously, the faster-rising thing will eventually overwhelm the slower-rising thing.

There’s quite a crisis of power. [ . . . ] The consensus at this point is that there will be a significant power shortfall next year. So, not like distant future. I believe the consensus estimate among analysts who follow the AI space very closely is that there will be at least a 15-gigawatt shortfall of power in 2027 for AI chips. The rate at which AI chips [are] being produced has been rising incredibly rapidly. They’re rising on the order of 40 to 50 per cent a year. But power available outside of China has been rising at 10 per cent to 20 per cent a year. Obviously, the faster-rising thing will eventually overwhelm the slower-rising thing.

Unusually for Musk, he makes a reasonable point but has lowballed the numbers.

Morgan Stanley analysts today revised their model for how much power would be needed to switch on the chips it projects will be sold. They now estimate, based on a total data-centre power requirement of 257 GW by 2028, a gap of between 30 and 40 per cent between US capacity and sales projections. Or to put it in layperson’s terms:

New York City demands 5.5-6 GW of baseload power: we are short power by six New York Cities.

New York City demands 5.5-6 GW of baseload power: we are short power by six New York Cities.

And that’s using their mid-estimate for innovative “time-to-power” solutions. Absent such mitigation measures, the estimate rises to 10 NYCs:

The MS team led by Stephen Byrd had previously estimated a US power supply shortfall of 38GW through 2028. Its new figures factor in a heavier draw from more complex servers built around Nvidia’s Vera Rubin and Rubin Ultra architectures, which threaten to increase the IT power demand by fivefold from 2025 to 2028:

Maybe this seems counterintuitive. Nvidia has said its Vera Rubin systems can deliver 30 times higher throughput per MW for agentic workloads than its predecessor.

Unfortunately for grid operators, the improved chip efficiency means data centre operators will improve margins by packing more processing power into the same footprint, Morgan Stanley assumes. System designers “use [GPU] efficiency gains to deploy more computing capacity rather than reduce electricity consumption”, it says.

Data centres have tended to host servers of eight GPUs. The next generation of integrated hardware comes in similar-sized boxes, but they draw more juice per installation and run a lot hotter. Nvidia’s Vera Rubin NVL72 server rack bundles together 72 GPUs with CPUs, network switches and hardware for power management and cooling in one miniaturised package. According to CoreWeave, a “typical server rack produces as much heat as 10-25 ovens running simultaneously”.

The unit economics of replacing legacy cards make sense. Switching from Nvidia’s Hopper family (released Q3 2022) to Rubin Ultra (due H2 2027) could lower the all-in cost of a processor operation by 94 per cent, the team estimates. Tokens per chip improve by approximately 15 times, so in spite of the higher running costs there’s still a sixfold improvement in tokens per watt.

But denser racks require “a step change in power demand” from 2027, Morgan Stanley says. It’s a textbook example of Jevons paradox, the phenomenon whereby improved efficiency makes a resource cheaper to use, which increases consumption and undoes the savings from individual efficiency.

Fenyman, Nvidia’s next next chip architecture, is scheduled for launch in the second half of 2028. It’ll be another step change, Morgan Stanley says. The same dynamics will increase US power demand year-on-year by a further 40 per cent in 2029, Byrd and team estimate. They’re then looking at a power shortfall of 107GW (or approximately 18 NYCs).

Per the first graph, “time to power” solutions could close the deficit. Bloom Energy’s fuel cells have the potential to add 5GW to supply by 2028, with a further 2GW coming via data centres being placed behind-the-meter in nuclear plants.

That still leaves a cumulative 17GW shortfall in Morgan Stanley’s mid case for 2028, which’d need to be filled by a raft of so-far-unannounced programmes to deliver gas turbines and off-grid generators that don’t yet exist in commercial forms. Spin the estimates forward a year and the vapourware in the numbers increases to a cumulative 24GW:

The way to play the theme is to invest in nearly any company that has a viable plan to make or sell energy, the bank says. These include “powered shell providers” (which we used to call bitcoin miners), novel generation companies and equipment suppliers.

This being a Morgan Stanley note, it also recommends a hit-and-hope buy of SpaceX. The telecoms company recently broke ground on a foundry in Texas to make its own gas turbine blades and vanes, and Musk spent an estimated $1bn of his own money in July to buy off-grid gas and diesel generator supplier APR Energy, it highlights, while cautioning that these are multiyear projects.

The argument presented is that AI is a virtuous circle. Each new generation of hardware will improve the unit economics of milling tokens, increasing margins for data centre operators and improving AI adoption rates. Improved adoption means more complicated workloads, which adds to infrastructure demands.

“However,” the team add in bold, “all of these advancements converge on one central issue: if we cannot close the power shortfall, the expected pace of improvement and adoption may not fully materialise in line with market expectations.”

They probably don’t need to spell out what happens to all those big tech off-balance-sheet guarantees in that scenario. Obviously, the faster-rising thing will eventually overwhelm the slower-rising thing.