LLVM for the Rest of Us

LLVM is everywhere. It sits underneath an enormous amount of modern software and touches everything from ordinary CPU compilation to GPU development, programming languages, machine learning systems, and increasingly the strange heterogeneous stack we are building underneath AI. Yet for something this foundational, I still find it surprisingly difficult to approach as an ordinary researcher. LLVM is over twenty years old, and trying to understand it for the first time can feel like arriving halfway through a conversation everyone else started years ago.

The problem is not a lack of information. If anything, LLVM’s maturity has created the opposite problem. There are countless tutorials, talks, reference manuals, source files, mailing-list discussions, books, and university courses. You can learn how to write LLVM IR, build a frontend, construct a pass, or dig into one of its backends. What I kept finding harder to answer was the question that should probably come before all of those: why does this machinery exist in the first place?

That question gets more interesting when you look at how computing has changed. Twenty-plus years ago compiler researchers were already wrestling with how to preserve a useful abstraction between software and changing hardware. Today we are surrounded by heterogeneous systems where CPUs, GPUs, accelerators, runtimes, compiler layers, memory hierarchies, and increasingly domain-specific representations all participate in turning an idea into something silicon can physically execute. If anything, the question of where software ends and hardware begins has become less obvious.

One of the earlier attempts to think about that boundary came from the 2003 paper LLVA: A Low-level Virtual Instruction Set Architecture. The question behind LLVA was simple enough to state: can we create a better boundary between software and hardware?

A traditional ISA, or Instruction Set Architecture, acts as a software-visible contract with the processor. x86-64 and ARM64 are examples of different ISAs. The ISA gives software a vocabulary of instructions and a set of rules the processor promises to follow. The LLVA authors identified an interesting tension here: the same hardware ISA was effectively being asked to serve as both the persistent low-level representation of software and the interface to a particular processor architecture.

LLVA explored separating those responsibilities. Instead of compiling a program directly toward the processor-facing ISA, software could target a V-ISA, or Virtual ISA. A processor-specific translator would then lower that representation toward an I-ISA, or Implementation ISA, understood by the underlying machine.

program
   ↓
V-ISA — Virtual ISA
   ↓
processor-specific translator
   ↓
I-ISA — Implementation ISA
   ↓
physical processor

The V-ISA would preserve information useful for describing and analyzing the program, while the translator and I-ISA would deal with the realities of a particular machine. Different processors could therefore use different translators and implementation ISAs underneath the same virtual representation.

There is an important detail here: the V-ISA was still supposed to be low-level. This was not an attempt to preserve every class, object, template, runtime system, or high-level abstraction from the language the programmer originally wrote. The representation still contained operations closer to loads, stores, arithmetic, branches, calls, and values. What it tried not to commit to too early were things like a fixed physical register set, stack-frame layout, low-level addressing quirks, limits on immediate constants, and other details that made more sense as properties of a particular machine.

The compromise LLVA was looking for was therefore something low-level enough to translate efficiently into machine code, while still being rich enough to preserve information a compiler could reason about. The larger dream of a persistent virtual ISA sitting between software and processors did not become the dominant role LLVM would eventually play, and I think that distinction matters. LLVA did not simply become modern LLVM exactly as it was proposed. What proved much more durable were some of the ideas surrounding representation itself.

That brings us to the 2004 paper, LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation, and this is where I think the story becomes much more relevant to the rest of us.

A compiler, in the most simplified sense, has a frontend responsible for understanding the source language, some kind of intermediate representation through which the compiler reasons about the program, and a backend responsible for eventually producing code for a target machine. GCC and other compilers already had sophisticated internal representations and optimization infrastructure before LLVM, so the important shift was not that LLVM invented the idea of having an IR between a frontend and backend.

The interesting idea was what could happen if a common, analyzable representation became persistent infrastructure rather than something treated mainly as an internal stage on the way toward machine code.

SOURCE LANGUAGE
      ↓
   FRONTEND
      ↓
      IR
      ↓
   BACKEND
      ↓
MACHINE CODE

An IR, or Intermediate Representation, is essentially a form of the program designed so that the compiler can reason about it. It is neither quite what the programmer originally wrote nor yet the final instructions a processor will execute. It occupies the middle, and that position turns out to matter enormously.

The title of the 2004 paper contains a word worth slowing down for: lifelong. The authors were referring to the lifetime of the software. Their idea was that the LLVM representation could remain useful beyond the first moment of compilation. Analysis and optimization could happen at compile time, link time, install time, runtime, and even between executions using information learned from the way the program was actually used.

SOURCE CODE
    │
    ▼
compile time
    │
    ▼
LLVM representation
    │
    ▼
link time
    │
    ▼
native machine code
    │
    ├───────────── LLVM representation preserved
    │
    ▼
runtime
    │
    ▼
profile actual behavior
    │
    ▼
idle time between runs
    │
    ▼
re-optimize
    │
    ▼
future execution

What I find so interesting about this is the change in attitude toward compilation. The compiler does not necessarily have to be a machine that takes source code, emits a binary, and then disappears from the story. A useful representation of the program can remain available long enough for the system to continue learning about and transforming that program later.

That sounds great until we run into the next problem. If the representation remains extremely high-level and tries to preserve classes, objects, inheritance, language-specific garbage collection, exception models, and every other abstraction programmers might use, then it becomes strongly attached to particular languages and runtimes. At the opposite end, if everything is immediately lowered into x86 or some similarly machine-specific representation, we gain proximity to the hardware but lose much of the structure that made the computation easier to analyze.

So once again, the interesting place is somewhere in the middle.

LLVM was designed as a low-level, language-independent representation while still retaining information useful to compiler analysis, including types, explicit control flow, explicit data flow, virtual values, loads, stores, calls, and a relatively small set of operations. This carries some of the same spirit I found interesting in LLVA: do not force the program to inherit every machine-specific decision before those decisions actually need to be made.

This is where SSA, or Static Single Assignment, becomes important. In ordinary code we might repeatedly reuse the same variable name:

x = 5
x = x + 1
x = x * 2

In SSA form, each produced value gets a new name:

x1 = 5
x2 = x1 + 1
x3 = x2 * 2

The reason this matters is not because compilers have some philosophical objection to mutation. It matters because the relationships between values become much easier to see. x2 has one definition, and x3 clearly depends on x2. The compiler can follow where values were created and where they are used without continually asking which assignment to a reused variable name is currently relevant.

LLVM also makes control flow explicit through basic blocks and control-flow graphs. If SSA helps answer where did this value come from?, the control-flow graph helps answer where can execution go from here? Put the two together and a program begins to look less like a flat list of instructions and more like a network of relationships the compiler can inspect.

That changed how I thought about IR.

LLVM IR is sometimes introduced as something resembling portable assembly, which is useful up to a point, but I think that description can make the important part easy to miss. It is not simply strange assembly that exists before the real assembly. It is a representation shaped so that important properties of the program remain visible enough for compiler transformations to reason about them.

Types preserve useful information about values. Loads and stores make interaction with memory explicit. Control-flow edges tell the compiler where execution may travel. SSA makes the origins and uses of values easier to follow. Operations such as getelementptr let LLVM express structured address calculations without immediately reducing everything to arbitrary byte offsets.

Once a program reaches this shared representation, compiler passes can begin asking questions of it. Can this calculation be eliminated? Can this function be inlined? Is this argument ever used? Is this value constant? Can this operation be moved? Can a value currently living in memory be promoted into an SSA value instead?

The important part is that those passes do not necessarily need to understand whether the program originally came from C, C++, or another source language. The frontend has already lowered those source-specific concepts into a common vocabulary the LLVM infrastructure can work with. That shared infrastructure is one of the ideas I think gets lost when LLVM is introduced by immediately throwing syntax at newbs. The power is not merely in writing something like %1 = add i32 %0, 1. The power is that different languages can eventually arrive at a representation where a large body of common compiler analysis and optimization can operate on them.

LLVM also established boundaries of its own, and that part is important because it prevents us from turning LLVM into something it was never intended to be. The 2004 paper explicitly says LLVM was not designed to be a universal compiler IR. It deliberately does not represent every high-level feature from every language, so the frontend must eventually lower those concepts into more primitive operations. At the same time, LLVM IR does not immediately encode every machine-specific implementation detail either.

It sits somewhere between those worlds:

language-specific meaning
          ↓
       FRONTEND
          ↓
       LLVM IR
          ↓
       BACKEND
          ↓
machine-specific details

Its goal is not to know everything. Its goal is to be useful at a particular level of reasoning.

This becomes especially interesting today because modern machine learning has made the limitations of lowering too early painfully obvious. A compiler looking at a matrix multiplication, tensor reduction, or convolution knows something useful about that computation. If those concepts are immediately reduced into individual loops, loads, stores, and scalar arithmetic, some of that information may be difficult or impossible to recover later when we want to optimize the workload for a GPU or another accelerator.

This is where MLIR comes in. Rather than assuming one intermediate representation should contain every useful abstraction, modern compiler infrastructure increasingly allows computation to move through multiple representations. A high-level representation can preserve information about tensors or domain-specific operations, another representation can expose structured loops or memory behavior, and eventually LLVM IR can take over once the computation has reached a level where its particular abstractions become useful.

 high-level ML operation
          ↓
domain-specific representation
          ↓
structured representation
          ↓
lower-level representation
          ↓
       LLVM IR
          ↓
target-specific lowering
          ↓
CPU / GPU / accelerator

The lesson becomes more nuanced than simply saying we should preserve information forever. We should preserve information while it remains useful, and lower when another representation becomes better suited to the next problem we are trying to solve.

This also brings LLVM directly into GPU programming, LLVM values can behave as though there are effectively unlimited virtual registers, but eventually a backend has to confront a physical machine with a very real and finite register file. At that point the abstraction begins meeting hardware reality.

        SSA values
             ↓
     register allocation
             ↓
physical CPU / GPU registers

This is where things like register pressure stop being abstract compiler vocabulary and start becoming performance constraints. Too many live values may require more physical registers, reduce occupancy on a GPU, or force values to spill into memory. The same broad transition happens on CPUs. At the LLVM level we can reason about abstract values, memory accesses, control flow, and transformations, but eventually the backend has to choose actual instructions, physical registers, schedules, calling conventions, and other details appropriate for x86, ARM, or whatever machine sits underneath it.

That is why I increasingly think LLVM matters far beyond people who intend to become compiler engineers.

We are entering a period where extraordinary amounts of computation are becoming available to increasingly ordinary people. Consumer machines can perform amounts of work that would have sounded ridiculous not very long ago, while the systems underneath modern AI increasingly depend on specialized parallel hardware. Putting petaflop-scale hardware within reach of ordinary people is only part of democratizing computation. If effectively using that hardware continues to depend on a conceptual stack that remains inaccessible to almost everyone outside a relatively small group of specialists, then we have democratized capacity without fully democratizing capability.

LLVM itself does not need to become simple. A system this mature and capable is going to contain real complexity, and pretending otherwise would probably make learning it harder rather than easier. What should become simpler is the path into understanding why that complexity exists.

We do not need every programmer to become a compiler engineer, but concepts like intermediate representation, SSA, lowering, optimization, data flow, control flow, code generation, register allocation, and the boundary between software intent and physical execution are becoming increasingly relevant to understanding modern computing, especially if we want to understand AI beyond the model.

At every boundary, something is being translated, every lowering step, decisions are being made about what information should survive and what can finally be discarded. At every level, the representation determines what the next part of the system is capable of seeing, and what it can see determines what it can safely transform or optimize.

That is what I want LLVM for the Rest of Us to explore. Not how to turn everyone into an LLVM contributor, but how to make the machinery between our ideas and the silicon beneath them understandable enough that more programmers can participate in the conversation.

There are already excellent technical resources for learning LLVM, including LLVM’s own My First Language Frontend with LLVM tutorial. Before asking how to build a compiler with LLVM, I want to understand why we needed something like LLVM in the first place, why this particular level of representation became so useful, and why the same questions around representation, lowering, and hardware boundaries have only become more important as CPUs, GPUs, accelerators, and machine learning systems continue to evolve.