Quadrillion Param Costs: KV Cache, Context Length, Frontier Margins

The models of 2028-2031 get much bigger than the models of 2026, going from 10T total params in 2026 to maybe 240T params in 2028 [1] and then 1.4 quadrillion params in 2031, as I estimate in the previous post from HBM bandwidth/capacity, scale-up system size, pretraining compute, and scaling laws. Yet as I show in this post, if the 240T 2028 model is priced at $14/$70 per 1M input/output tokens (1.4x the API price of Mythos 5), it's going to have a 70% gross margin, and the same holds for the 1,400T 2031 model when priced at $30/$150. Cutting the price in half to $15/$75 per 1M input/output tokens lowers the gross margin to 40%, which seems painful but survivable. Going in the other direction, doubling the price to $60/$300 allows serving requests with up to 3M tokens of context at the same 70% gross margin.

These prices rest on token costs that I calculate from first principles in this post, using estimates of future hardware specs and costs. I link the 10T 2026 model to Mythos 5 to compare with its actual API prices, also performing the calculations for my guess about Opus 4.8, and the result is gross margins of 83-89%, consistent with SemiAnalysis claims I cite in that section of the post. This is despite my Opus 4.8 guess having 650B active params, and my Mythos 5 guess having 1.3T active params, much more than any open weight model (which mostly have about 50B active params). This also predicts that Anthropic is currently using about 0.5-1 GW of inference compute (that brings in revenue at the API gross margins).

The cost estimate for the hypothetical 240T 2028 model (7.9T active params) is merely $2.5/$46 per 1M input/output tokens, and for the 1,400T 2031 model (48T active params) it's $12.3/$67. The cost of output tokens barely increases as a result of how KV cache size per token scales with the number of active params, which makes longer contexts cheaper and keeps the overall cost per average token down. These costs assume long-term rental prices…