Why AI Models Hallucinate

In September 2025, OpenAI published a paper that said something the AI industry already suspected but hadn’t quite articulated. The paper, “Why Language Models Hallucinate”, authored by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang, didn’t just catalogue the problem. It pointed the finger at the evaluation systems that are supposed to keep models honest and argued that those systems are actively making hallucination worse.

The paper’s central argument is disarmingly simple. Language models hallucinate because we reward them for guessing. The training loops, the benchmarks, the leaderboards that determine which model gets called “best” all operate on a scoring system that treats confident wrong answers and honest uncertainty as equally worthless. Under those rules, the rational strategy for any model is to always take a shot, even when the evidence is thin. And that strategy produces hallucinations.

Researchers have known for years that models tend toward overconfidence. But the OpenAI paper formalised it with mathematical precision and made an argument that goes further than most. The problem is that our entire evaluation infrastructure systematically incentivises the specific failure mode we claim to care most about fixing.

The Mechanics of Making Things Up

To understand why the paper matters, it helps to start with what hallucination actually is at a mechanical level.

During pretraining, a language model learns to predict the next token in a sequence. It ingests billions of documents and builds a statistical model of what words tend to follow other words in what contexts. This process is extraordinarily powerful for capturing patterns, grammar, reasoning structures, and factual associations. But it has an inherent limitation that no amount of scale can fully overcome.

Some facts appear in training data frequently enough that the model can learn them reliably. The capital of France, the boiling point of water, the year the Berlin Wall fell…