Small Is Beautiful

Estimates are that, to justify the AI platforms' enormous capex plans, by 2030 they need to be generating around $2T/year in revenue. If every adult resident of the US spent $20/month on AI, it would generate $68.5B/year. Clearly, only the enterprise market stands even a remote possibility of generating the bulk of the $2T.

There are three major threats to the prospect of AI platforms extracting 6% of current US GDP from the enterprise market, and thus to OpenAI's and Anthropic's ambitions to IPO in the near future. First, faced with AI's Affordability Crisis, companies have been placing strict limits on employees' spending on AI tokens.

SourceUS Lead in the AI Race With China Is Rapidly Narrowing:

Bloomberg in partnership with researchers at Vals AI, an independent AI evaluation and benchmarking platform, tested seven models from frontier Chinese and US companies to see how they performed in a real-world task. They were asked to create a fictional coffee e-commerce site called Brewberg using the same prompts. Most of the models scored 100% functional accuracy despite occasional design misses, but with very different price tags. The experiment employed the top performing models in July from Anthropic and all the Chinese firms, as well as more affordable models from OpenAI and Google.

They all did reasonably well, but the two best were Claude Fable 5 at $48.99 and Kimi K3 at $11.99. Chinese models charging much less for almost the same performance are grabbing market share:

the use of Chinese models overtook US platforms globally for the first time in June, and accounted for more than 60% of market share last month, on OpenRouter, a tech platform that offers software developers access to hundreds of AI models. It is a widely watched gauge of model usage despite tracking just a fraction of global AI consumption. The US, parts of Europe and Asia now favor Chinese labs, according to the same data.

On Hugging Face, Chinese AI models account for 41.4% of generative model downloads among developers, 5 percentage points higher than US models.

Third, it isn't just that the Chinese models are cheaper to run remotely, but also that because they are open-weight they can be run on affordable in-house systems, which means that:

• They don't give Donald Trump a kill-switch for your busines.
• They don't give Sam Altman or Dario Amodei a kill-switch for your busines.
• They don't require giving the Chinese, Sam Altman or Dario Amodei all your business' critical data.
• They provide visibility into and control over AI costs.
• They are even cheaper.

The question is "compared to the closed-weight US models, what do you lose by running open-weight models in-house?" Below the fold I discuss a major study from Stanford that answers the question.

The 37-page paper is Intelligence per Watt: Measuring Intelligence Efficiency of Local AI by Jon Saad-Falcon et 14 al. Their abstract reads:

Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Demand growth strains this paradigm faster than providers can scale. Two advances create an opportunity to rethink it: small, local LMs (<=20B active parameters) now achieve competitive performance to frontier models on many tasks, and local accelerators (e.g., Apple M4 Max) can host these models at interactive latencies. This raises the question: can local inference viably redistribute demand from centralized infrastructure? This requires measuring both whether local LMs can accurately answer real-world queries and whether they can do so efficiently on power-constrained devices (e.g., laptops). We propose intelligence per watt (IPW), task accuracy per unit of power, as a unified metric for the capability and efficiency of local inference across model-accelerator configurations. We evaluate 20+ state-of-the-art local LMs, 8 hardware accelerators (local and cloud), and 1M real-world single-turn chat and reasoning queries. For each query, we measure accuracy (local LM win rate against frontier models), energy, latency, and power. We find three key results. First, local LMs successfully answer 88.7% of these queries, with accuracy varying by domain. Second, longitudinal analysis from 2023-2025 shows IPW improved 5.3x, driven by both algorithmic and accelerator advances, with locally-serviceable query coverage rising from 23.2% to 71.3%. Third, local accelerators achieve at least 1.4x lower IPW than cloud accelerators running identical models, revealing significant headroom for local accelerator optimization. These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure for a substantial subset of queries, with IPW serving as the critical metric for tracking this transition.

Joachim Klement provides an introduction to this "Small Language Models (SLMs)" paper in If this is true, the hyperscalers are toast:

First, they ran a series of SLMs (QWEN 3, GEMMA 3, GPT-OSS, GRANITE 4.0) that can be downloaded on a local PC and compared their performance with cloud-based state-of-the-art LLMs (ChatGPT 5, Claude Sonnet 4.5, Gemini 2.5 Pro).

They ran these SLMs on local PCs powered either by an Nvidia chip or an Apple M4 chip, as they are readily available in current high-end desktop computers ...

Then they traced the performance of these SLMs vs LLM between 2023 and October 2025 on both chat tasks and reasoning tasks.

Saad-Falcon Fig 2caption is:

Local Models Rival Cloud Models Across Diverse Benchmarks: Individual model performance scales with size, ranging from 31.5–69.4% for IBM GRANITE 4-H-S MALL, 30.0–83.6% for GEMMA 3-12B, 51.5–80.4% for GPT-OSS-120B, and 66.5–89.5% for GEMINI 2.5 PRO . Local routing (best local LM per query) achieves 97.8%, 88.3%, 77.0%, and 92.4% on WILDCHAT, NATURAL REASONING, SUPER GPQA, and MMLUPRO respectively, sur- passing cloud routing (100%, 82.9%, 66.5%, 87.4%) on three of four benchmarks.

Klement Fig 1reformats the results into charts. First for chat tasks:

which still make up the vast majority of requests today. As you can see, in every domain, the best SLM is able to find the same or better answers than an LLM in 90% or more of the cases, with an average across all domains of 98.6%.

It is really hard to justify spending 4-6 times as much for a 1.4% improvement in performance, so LLMs are no longer really necesssary for chat tasks.

Klement Fig 2reasoning tasks:

which are obviously more demanding, SLMs are catching up fast. On average, they provide a better or at least as good an answer as LLMs in 62.5% of the cases.

SLMs may be catching up fast on reasoning tasks (see IBM's Granite 4.2 below) but it will be a while before LLMs are obsolete for these tasks.

Klement Fig 3Klement points out:

the tasks for SLMs and LLMs are typically a mix of chat requests and reasoning tasks, so the third chart shows the weighted average of chat request performance and reasoning performance based on the frequency of tasks in each domain. As you can see, on average, SLMs are as good if not better than LLMs in 81.2% of the cases, with the LLMs having a significant advantage only in areas like engineering, life sciences, transportation and computer sciences.

But it’s not just accuracy. SLMs achieve this performance at energy and compute costs that are between 50% and 85% lower than for an LLM, depending on the SLM and hardware used in the computer.

Figure 7reasoning tasks:

For reasoning tasks ... the pattern differs substantially. While levels 1-3 show strong improvements (+24.0, +37.8, and +53.9 pp respectively), levels 4 and 5 exhibit markedly slower progress. Level 4 improves by only +23.8 pp (7.93% to 31.72%), and level 5 remains largely unsolved with just +1.5 pp improvement (3.27% to 4.72%). This suggests that while local models have rapidly closed the gap on moderately difficult reasoning tasks, the hardest reasoning problems (those requiring either massive scale or capabilities beyond current architectures) remain a significant frontier. The presence of 134 level 5 problems (16.5% of the reasoning dataset) that remain 95% unsolved indicates substantial headroom for future model development in complex reasoning domains.

To understand why the rate at which SLMs catch up is critical we need to study Groundbreaker's The Teaser Period: Why the AI Boom Is Built to Break, which starts from the analogy between the subprime crisis and the AI bubble:

Paulson & Co. laid out the arithmetic that same month in a comment letter to the FDIC: Over 80% of recent subprime originations, it observed, were two- or three-year adjustable-rate products. The average subprime borrower’s mortgage payments already consumed roughly 40% of their gross income at the teaser rate. Almost none of them could service the reset rate out of income.

The crisis, in other words, was written in advance by the instruments themselves. The market looked at the reset wall and kept buying, because every participant believed the exit would arrive before the reset: home prices would keep appreciating and the borrower would refinance into a fresh teaser before the old one expired.

Sourcesubprime Adjustable Rate Mortgage (ARM):

The take-or-pay compute contract - the instrument at the center of the AI build-out - has a structural feature that almost no one prices: its payments do not begin at signing. They begin at delivery. A lab signs a multi-year capacity commitment today, but the payments do not start until the data center is energized, the capacity is accepted, and the contractual ramp schedule commences - an interval set not by finance, but by construction: siting, powering, and filling a gigawatt-scale campus takes 24-to-36 months from signature - mirroring the two-to-three-year teaser of a subprime ARM.

More than $2.3 trillion of compute contracts now sit on the books of the four largest American cloud providers as remaining performance obligations and contracted backlog - signed, celebrated, capitalized into equity prices, and, critically, not yet billing.
...
The parallel to 2006 is exact and it explains the single most-cited absurdity of this cycle: How does OpenAI, a company with some $40 billion of run-rate revenue, sign $1.4 trillion of compute commitments? The same way a household with $60,000 of income signed a $600,000 mortgage: because the terms at signing do not require the payment yet, and because everyone at the table - borrower, lender, and the market - believes the growth will arrive before the payment does.

The chart shows that, in 2027 and 2028, the AI platforms will need to shell out $852B in cash for compute, whether they use it or not. We don't know how much revenue they are currently generating, but they want us to believe it is in the region of $100B. Ignoring all their other costs, they have to increase their revenue more than 4x next year to cover their contractual payments for compute. That means they have to extract at least $400B from the enterprise market in return for supplying it with technology that is slightly better than technology companies can run in-house around an order of magnitude cheaper.

What matters isn't the relative price/performance of in-house SLMs versus remote LLMs now, it is their relative price/performance when the LLMs' contractual compute payments come due, i.e. next year. It seems very unlikely that companies already balking at the cost of the AI platforms' products by moving to Chinese models will increase their spend 4x next year. It seems equally unlikely that investors will give the AI platforms a few hundred million dollars next year to burn so as to postpone the day of reckoning by another 12 months.

The Stanford study collected data in October 2025. Developments since, with more to come, have already significantly increased the price/performance advantage of SLMs. The include:

• Nvidia's RTX Spark "superchip", disucssed by @pramodchandrayan in NVIDIA Just Put a 120-Billion-Parameter AI Model in Your Laptop. Here’s What That Actually Changes.:
  

  At Computex 2026 in Taipei on June 1st, CEO Jensen Huang announced the RTX Spark superchip — a single piece of silicon that combines a 20-core Arm CPU, a Blackwell GPU with 6,144 CUDA cores, and 128 gigabytes of unified memory, connected by NVIDIA’s NVLink chip-to-chip interconnect. The whole package delivers up to one petaflop of AI compute in a laptop form factor.
  
  The number that matters: RTX Spark can run a 120-billion-parameter language model entirely locally, with a context window of one million tokens, without a single byte leaving your machine.

  This is the guts of Nvidia's $5.2K DGX Spark desktop. It was Portent #20
• The Exxact Valence Nvidia DGX Station discussed by Michal Malewicz in NVIDIA just killed big AI and… You’re the winner?:
  

  Here’s a $94,011.50 desktop computer for sale right now.
  
  It’s not a server rack or some crazy cloud infrastructure monstrosity.
  
  It’s a (rather big) tower PC. Kinda like the one you played Cyberpunk 2077 on.
  
  It sits under a desk and plugs into a wall like a regular desktop. The only difference is that it runs trillion-parameter AI models with no API keys, no per token payment, and no personal data leaving the room.

  It is 18 times as expensive as the DGX Spark but can run models more than 8 times bigger. This was Portent 31.
• Michael Nuñez' Perplexity partners with Nvidia to launch Portable Computer, a fully local AI agent with zero token costs:

  Perplexity is launching Portable Computer today, a version of its agentic "Computer" platform that runs entirely on hardware users already own — starting with Nvidia's DGX Spark desktop supercomputer and Linux machines equipped with Nvidia RTX GPUs.
  
  The launch, developed in close partnership with Nvidia, is one of the most aggressive attempts yet to move serious AI agent workloads off the cloud and onto local devices. The model, the user's files, and the work itself can all stay on the machine. Work completed locally consumes no billing credits, and the company says every task starts on the device by default — with the system asking permission before sending any individual step to a more powerful frontier model in the cloud.

  This was Portent 45.
• Apple's announcement of the latest Mac hardware, discussed by Samuel Axon in Apple’s new desktop computers are designed specifically for local AI development:
  

  Apple announced new iterations of both desktops, along with two new chips: the M6, the first 2nm chip in Apple’s M-series lineup for Macs, and the M5 Ultra, now the most powerful chip in the lineup for most things—especially AI workloads.
  
  There aren’t any major new features for either machine. This is just a specs bump. But based on how Apple is presenting these refreshes, they’re leaning hard into those use cases, which weren’t even a thought when earlier iterations were first engineered.
  
  The devices’ popularity for production inference took off after macOS 26.2 shipped last December. According to Apple’s release notes, 26.2 enabled “low-latency communication between Thunderbolt 5 hosts for use cases including distributed AI inference using MLX.” Thunderbolt 5 is a very fast wired data connection, and MLX is an open source array framework designed to help machine learning workflows take full advantage of the M-series chips’ unified memory.
  
  Since then, both hobbyists and professional developers and researchers have been essentially daisy-chaining Mac minis or Mac Studios to run inference on local large language models that are much bigger than anything that could run a single mass-market device—providing an alternative to ultra-beefy specialized hardware featuring specialized Nvidia GPUs.
  

  This was Portent 46.
• IBM's latest model release, discussed by Samuel Axon in IBM’s new Granite 4.2 models ride the wave of interest in local LLMs:
  

  IBM has rolled out the newest models in its family of open-weight large language models designed to be downloaded and self-hosted. The newly launched Granite 4.2 comes in 3B, 8B, and 30B parameter variants.
  
  Like previous versions, IBM is taking a decoder-only approach here. These new releases offer a 128,000-token context window natively. The 8B and 30B variants (not the 3B one) also go through an agentic reinforcement-learning block; they were trained for expanded capabilities like using the terminal, searching the web, or using external tools. The 3B model supports tools too, but without the same level of specialized training.
  
  Beyond those tweaks, this release is particularly notable because, as IBM itself writes, “Granite 4.2 is the reasoning-focused release of the Granite language-model family.

  This was Portent 47.

Consider not just what Ding et al's graph of the narrowing performance gap between closed= and open-weight models, but also what Saad-Falcon et al's Figure 7 will look like after another year of both hardware and software developments like these.

Saad-Falcon et al argue that it isn't just the raw price/performance that advantages local compute:

System-level benefits offset per-query efficiency disadvantages. While cloud accelerators demonstrate 1.4× to 7.4× higher intelligence efficiency per query, local deployment provides complementary system-level benefits that offset this disadvantage. Local inference avoids datacenter infrastructure costs, network latency, and API pricing, while enabling 88.7% of queries that local models can handle correctly to bypass cloud compute entirely. As demon- strated in Section 4.3, intelligent routing between local and cloud infrastructure can achieve 60–80% reductions in total energy, compute, and cost compared to cloud-only deployment, even when local accelerators are individually less efficient. These findings suggest that the path to efficient AI infrastructure lies not in local accelerators matching cloud efficiency, but in routing systems that leverage the complementary strengths of both paradigms: local processing for the majority of straightforward queries and cloud infrastructure for the minority requiring frontier model capabilities.

It looks increasingly as though Klement is right that the hyperscalers are toast, because the vast majority of inference will happen locally while the massive data centers will be used only for training and for inference on massive models so expensive that almost no-one can afford them.