Between the Graph and the Silicon: Inside the Apple Neural Engine Compiler

As I continue my journey into the abyss of the compiler, one idea has finally started to settle in for me, a compiler is not just a translator that takes language A and turns it into machine B. That description is useful when you are first learning, but it leaves out most of what makes compilers interesting. A compiler is also a system for managing information while gradually changing the form of a program. At every stage, it has to decide what still matters, what can be transformed, and what can finally be discarded.

That was a large part of what interested me while writing about LLVM, and it followed me directly into KGEN and MLIR. LLVM pushed me to think about why an intermediate representation exists in the first place. KGEN complicated that picture further by showing why one representation may not always be enough. Mojo can preserve source-level information in LIT while that information is still useful, move into KGEN while parametric information still matters, and retain structured control flow through HLCF until the compiler has finished reasoning about it.

Much of that discussion revolved around what the compiler wants to preserve or transform. My next venture, Spencer Bryngelson’s Apple Neural Engine: Architecture, Programming, and Performance, introduces something a little less negotiable, the machine itself.

The Apple Neural Engine is an unusually opaque target, which, given that this is Apple, should probably surprise nobody. Apple publicly exposes the ANE through Core ML, while the compiler, program format, driver, firmware, and much of the machinery underneath it remain undocumented. Bryngelson’s paper is an independent reverse-engineered account of that stack, based on direct measurements and analysis of Apple’s private software. For this piece I am mostly interested in what follows the compiler as a neural-network graph is transformed into something the ANE can actually execute.

Consider a neural-network graph consisting of a convolution, followed by a bias addition, followed by a ReLU activation. At the graph level, these are three perfectly reasonable operations. Perform the convolution, add the bias to its result, then apply the activation. When I first started reading the compiler chapter, I more or less assumed that some progressively lower-level version of those three operations would eventually become three pieces of work for the accelerator.

That assumption was short lived.

The ANE compiler can absorb the bias and activation into the convolution itself. Padding can be folded into it as well, and parts of the quantization path can disappear into the same backend operation. The convolution’s bias becomes part of the hardware’s gain-offset machinery, while the activation can occupy one of the activation slots associated with the operation. What entered the compiler as conv → bias → relu can therefore leave this stage as one fused ANE operation rather than three independent dispatches.

This connected immediately to what I had been learning from KGEN, except from the opposite direction. With KGEN, I always landed on why a compiler would preserve a concept. Here I found myself asking when the distinction between two concepts stops being useful.

A convolution, bias, and ReLU remain useful as separate ideas when I am reasoning about the structure of the neural network. Once the compiler reaches hardware that already has a place to express the bias and activation as part of the same execution path, preserving those boundaries may no longer buy us anything. Neither representation is somehow more truthful than the other.

Now our four broad phases are as followed: fusion, legalization, scheduling and task-descriptor partitioning, followed by memory and DMA optimization. Put into less compiler-shaped language, the sequence is fairly intuitive. The compiler first has to understand what computation was requested and whether several pieces of it can become one machine operation. It then has to determine whether the resulting operation is actually legal on this hardware, decide how the surviving work should be ordered and divided, and finally decide where the data should live and how it should move while all of that happens.

Fusion and the usefulness of boundaries

Fusion is a good place to begin because it makes the relationship between representation and hardware unusually concrete.

The network representation entering this part of the compiler can still contain the convolution, bias, and activation as independent operations. The fusion happens later inside the ANE compiler’s own graph machinery. That distinction matters because it shows that the original boundaries were not mistakes. They were useful at one stage and unnecessary at another.

The same logic applies to other operations. Transposes can sometimes disappear into neighboring work or use dedicated machinery instead of becoming separate dispatches. Scaling and batch normalization can be absorbed into the same gain-offset path. A dequantization step can be folded into the convolution’s weight path rather than surviving as an operation of its own. The compiler is continuously asking whether an intermediate result or operation boundary still represents something the target actually needs to see.

There are limits. An addition between a convolution result and a constant bias fits naturally into the gain-offset machinery because one side of the operation is constant. An addition between two independently computed convolution outputs does not. Both operands are live values, so the addition remains a real backend operation. Concat can create another fusion boundary, attention introduces harder segment boundaries, and later capacity constraints can even force the compiler to split something it had previously fused.

I like this nuance because it prevents fusion from becoming another vague “compiler optimization makes things faster” explanation. The compiler is not combining operations simply because fewer operations sound better. It is combining them when the hardware provides a representation capable of expressing their combined behavior, and keeping them separate when it does not.

When a valid computation is not a legal machine program

An intermediate representation can describe a perfectly valid computation. The mathematics can make sense, the graph can be well formed, and the compiler frontend can understand exactly what you are asking for. None of that guarantees that this particular machine can execute the computation in that form.

The ANE has an envelope. Tensor rank is limited, individual dimensions have bounds, convolution kernels have target-specific constraints, and coefficients have to fit within particular memory limits. If an operation falls outside those boundaries, the compiler has to tile it, split it, transform it, or break the graph into additional segments.

Its sounds straight forward, but I think it exposes a distinction that is easy to lose when people say a piece of hardware “supports” some operation. Representable, valid, lowerable, and executable are not necessarily the same claim.

Now, The compiler exposes validators that check whether operations satisfy expected shapes, types, dimensions, operand counts, and target-specific capabilities. Yet passing one of those validators does not necessarily mean backend code generation will succeed. On the M1, the paper found operations such as top-k, sort, dynamic slice, and 3D convolution that can pass parts of that validation surface and still fail during backend lowering.

That means saying “the ANE supports this” . But, supported where?

Can the intermediate language represent it? Does the frontend accept it? Does the validator approve it? Can the backend turn it into ANE operations? Does the resulting program actually execute on this generation of silicon?

Those are different boundaries, and the farther down the stack we travel, the less useful the word supported becomes without saying which one we mean.

When matrix multiplication decides to become a convolution

One of my favorite examples in the chapter involves matrix multiplication because it makes another compiler instinct I had been carrying around fall apart.

If a matrix multiply’s right-hand weight fits within the ANE’s on-chip working-set budget, the compiler can rewrite that matrix multiplication as a resident convolution. If the weight is too large, it remains a tiled matrix multiply. On the M1, the working-set threshold involved here is about 2 MB.

The first time I read that, part of my brain still wanted to object that I had asked for a matrix multiplication, not a convolution.

The compiler is responsible for preserving the computation, not the name I originally gave the operation. “Matrix multiplication” may be the best representation for understanding the mathematics, while “resident convolution” may be a better realization on this particular accelerator when the weights fit comfortably on-chip.

This is where lowering starts feeling less like translation and more like selecting among possible realizations of the same intent. The source-level operation is not sacred. What matters is whether the transformation preserves what the program is supposed to compute while making better use of the machine underneath it.

The machine has started influencing not only how an operation executes, but which representation of that operation is useful at all.

Eventually bytes start making compiler decisions

Once the graph has been fused and legalized, the compiler still has to determine how the resulting work should actually run. Chapter 22’s scheduling phase linearizes the operation graph and divides that schedule into task-descriptor partitions that fit the engine’s on-chip working-set budget.

The M1 makes this wonderfully concrete because the scheduler is working against an approximately 2 MB static-memory ceiling. Rather than dividing work according to some arbitrary number of operations, the compiler tentatively adds another layer, estimates the peak live memory pressure across its lifetime, and closes the current partition when the next piece of work would push that pressure beyond the allowed budget.

This might be the part of the chapter that made the whole thing click most clearly for me.

Higher in the compiler I can spend all day discussing representations, graph structure, fusion rules, and operations. Eventually the decision about where a program gets divided can come down to a much less philosophical approach, does the live data still fit?

At that point, information is no longer just an abstraction the compiler reasons about. It occupies bytes, and those bytes need somewhere to exist.

That realization slightly changes my definition of a compiler again. It is certainly managing information, but as we get closer to the machine it increasingly becomes a resource-management system as well. The compiler has to think about how much fast storage is available, which values are alive at the same time, and whether the chosen execution order will keep the working set within that physical limit.

The arrow between two operations is not free

DMA, or Direct Memory Access, is essentially part of the machinery that allows data to move between memory and the accelerator without requiring the CPU to personally shepherd every byte along the way.

At a high level, I might draw two operations as A → B and think mostly about the dependency: B needs whatever A produces.

Closer to the hardware, that arrow starts acquiring baggage. Where did A place its result? Does it need to be written back out to dynamic memory? Does B have to read it again? Can the result remain on-chip? Are the two operations scheduled next to each other? Does their tile geometry line up? Does anyone else need that value before it can be reused?

Chapter 22 describes several allocation states that distinguish data kept resident on-chip, streamed from dynamic memory, held in ring buffers, rewritten in place, or chained directly from one operation into the next. Under the right conditions, a producer with one consumer can leave its result on-chip so the following operation consumes it directly rather than forcing an unnecessary trip out to dynamic memory and back.

That is an easy detail to overlook if you live entirely at the graph level. The arrow looked free there. Down here it has become memory traffic.

The compiler also rearranges destination-buffer strides to avoid bank conflicts, folds padding, and packs weight streams. These decisions are not as glamorous as inventing an IR or proving an optimization correct, but they expose something fundamental about accelerator programming: once computation becomes cheap enough, moving the data required by that computation can become a large part of the problem.

Again, the abstraction changes because the question changes. The neural-network graph cares about dependencies between operations. The memory optimizer has to care about where the resulting bytes are physically available when those dependencies are satisfied.

Representation has a cost

While I was working through this, Chris Lattner added a useful refinement to something I had been trying to articulate in my KGEN piece. I had become fascinated by MLIR’s ability to let different representations preserve the abstractions that matter at different phases. His point was that every additional IR also adds complexity, and that complexity has to justify itself. The goal is not to preserve as many representations as possible, but to use the fewest that still express the core abstractions needed at each stage.

That fits surprisingly well with what I see happening in the ANE compiler.

The network graph keeps convolution, bias, and activation separate while that distinction is useful. The fused representation gives them up when the hardware can express their behavior together. Legalization preserves enough shape and capability information to determine whether the resulting work belongs inside the machine’s envelope. The scheduler increasingly cares about live ranges and memory pressure, while the allocator eventually cares about residency, strides, tiles, and movement.

The compiler does not preserve an abstraction because abstractions are nice to have. It preserves one because there is still a question that is easier to answer while that abstraction exists.

Once the question has been answered, carrying that representation farther may become unnecessary overhead.

This is the part that has gradually changed how I think about lowering. I used to imagine it primarily as a movement from something high-level toward something lower-level. I still think that is directionally true, but it misses why the program keeps changing shape.

When representation answers to physics

Looking back across LLVM, KGEN, and now the ANE compiler, I think I have accidentally been following the same problem farther and farther down the stack.

With LLVM, I wanted to understand why an intermediate representation mattered at all. A useful IR preserves enough structure that the compiler can continue reasoning about a program rather than committing immediately to the limitations of one machine.

KGEN and MLIR expanded that idea. Different stages of compilation may require different useful abstractions, and a compiler can move among them as the information it needs changes. The interesting problem becomes deciding what still deserves to survive and when it is safe to let something go.

The ANE introduces another constraint because eventually those representations encounter a physical machine with finite memory, particular execution units, supported and unsupported forms, target-specific limits, and actual costs for moving data around.

A convolution, bias, and ReLU can collapse because this accelerator has machinery capable of expressing their combined behavior. A matrix multiply can become a convolution because that realization happens to fit the target better under a particular memory constraint. A graph can be partitioned in a place that has nothing to do with the way the programmer organized it because the next live tensor would push the working set beyond what the accelerator can hold. An intermediate value can remain resident or be streamed according to whether the surrounding execution makes that possible.

The computation still has to mean the same thing in the sense that ultimately matters to us: given the same valid inputs, it should produce the result we asked for. But nearly everything about how that computation is represented can change on its way down.

They look less and less like a machine for mechanically translating one language into another and more like the place where several descriptions of the same computation negotiate with one another. The programmer has one description. The model graph has another. The compiler creates several more. Eventually the accelerator introduces constraints that none of those previous representations can simply ignore.

With a network description and ends with a loadable program for the ANE. What happens in between is fusion, rewriting, scheduling, partitioning, allocation, and data movement, but underneath all of those mechanisms is a fairly straight forward progression. The compiler keeps changing the program until the way we describe the computation is compatible with the way the machine can physically perform it.