Prompt Caching

Large language models are often thought of like functions: send in some text, receive some text. That is a useful abstraction, but it ignores one of the most important parts of running a coding agent: most of the input is the same as last time. In other words we mostly append to it.

A coding agent sends the model its system prompt, tool definitions, project instructions, conversation history, tool calls, and tool results. On the next turn it sends almost all of that again, plus a small amount of new material. Once a session has grown to tens or hundreds of thousands of tokens, recomputing the whole prompt for every turn is slow and expensive.

Prompt caching is what makes this somewhat economic, but it is also quite fragile. A changed tool definition, a model switch or a provider routing decision can turn what one would expect to be a cheap incremental request into a full replay of the context.

For coding agents, cache behavior is therefore not just an implementation detail or optimization. It affects latency, cost, tool design, session design, and even which product features should be made available.

What a KV Cache Contains

A transformer processes a prompt in two broad phases. During prefill, it reads the input tokens and computes attention state for them. During decode, it produces new tokens one at a time.

At each attention layer, every processed token produces a key and a value. These are not quite like key-value lookups in a hash table: both are arrays of numbers, usually floats or lower-precision quantized values. When processing a new token, the model compares that token's query with the earlier keys to determine how relevant each earlier token is. It then uses those relevance scores to form a weighted mixture of the corresponding values. In that sense, a key is what the model matches against, while a value is the information it retrieves (but the lookup is fuzzy rather than "returning a single exact match" like a dictionary lookup.)

Those keys and values are retained so that the next generated token can attend to everything that came before without recomputing the earlier tokens. This retained state is the KV cache.

Conceptually, a request looks like this:

request 1:

[system][tools][user][assistant][tool result][user]
<--------------------- prefill -------------------->
                       |
                       K and V tensors per token and layer

request 2:

[system][tools][user][assistant][tool result][user][new]
<---------------- reusable prefix ----------------><--->
                                                    |
                                                    new work

The real representations are more complicated, model-specific, and "quite" large. The important property is that they correspond to a particular token prefix. Two prompts that mean the same thing but tokenize differently do not share a KV cache. If a token changes in the middle, everything after that token is a different continuation.

Prompt caching extends the lifetime of this state beyond one generation. When the next API request from the coding agent begins with the same tokens, the inference system can reuse the stored work for the matching prefix and prefill only the new suffix. So far, the theory.

Where the Cache Lives

In order for a cache to work it needs to be stored somewhere, and it needs to be addressable. There are two broad ways inference systems make KV caches available to a later request.

The simpler approach is session affinity. It works by keeping the KV cache on or near the GPU that computed it, and routing the next request back to the same worker. A session ID or prompt-cache key becomes a trivial routing hint and so you can potentially even deal with this problem on the HTTP load balancer level without having to look into the payload.

request(session-42) --> router --> worker 7 --> GPU 7 KV cache
next(session-42)    --> router --> worker 7 --> GPU 7 KV cache

This avoids moving a very large cache over the network. It is fast when it works, but it constrains scheduling. The selected worker can become overloaded, restart, or evict the entry. A router may also decide that balancing the fleet is more important than preserving one session's cache. It is however a very attractive solution because it works with little extra deployed infrastructure and hardware.

The other approach is to distribute the cache. KV blocks can be stored in another memory tier or made available across workers, so a request is not tied as tightly to one GPU.

                         +--------------------+
request --> scheduler -->| worker 3 / GPU 3   |
             |           +--------------------+
             |
             +----------> distributed KV blocks
             |
             +----------> worker 9 / GPU 9

That improves scheduling flexibility and recovery, but moving, indexing, and retaining KV blocks is itself a systems problem. Implementations mix GPU memory, host memory, local storage, remote storage, prefix-aware routing, and eviction policies in different ways.

To put KV caches into perspective: they can be large but they are in some ways smaller than one would assume. With various tricks, the size of KV caches can be reduced to a handful of gigabytes, even for long conversations.

Caches and Prefixes

Pi sessions are trees, not lists. /tree can move the active conversation back to an earlier point and continue along another branch. A rewind can discard the active suffix without deleting it from the session file. A new branch can share most of the old context, a little of it, or effectively none of it. This design is not unique to Pi, quite a few coding agents have something at least conceptually similar. Even if you do not represent the session as a tree, it's not uncommon for agents to have some form of rewinding.

                             +-- E -- F  another branch
                             |
session S: root -- A -- B -- C -- D  current branch
                   |
                   +-- Z  branch near the start

All three branches can have the same Pi session ID. From the router's perspective they are one session. From the prompt cache's perspective they are three token sequences with only partial prefix overlap.

If the cache keeps reusable prefix blocks, jumping from D to F may still reuse root -> C. If it only retains the hottest continuation, if the shared blocks were evicted, or if the request is routed elsewhere, the hit can be much smaller. Jumping to Z may preserve only the system prompt and initial tool definitions even though it starts from A. The precise cache management behavior here depends greatly on the providers.

The reverse can also happen. /fork or a new session can produce a new session ID while carrying over a large amount of identical context. A routing system that isolates caches by session key may fail to notice that useful overlap.

The reusable prefix determines what work can be cached. Session identity merely helps the infrastructure find likely content. On some systems the routing key is crucial to manage caches, on others it's merely an optimization.

Explicit vs Automatic Prefix Caching

Provider APIs expose caching in two main styles.

Anthropic's traditional interface uses explicit cache_control points. The client marks boundaries after stable parts of the request, such as the system prompt, tool definitions, or the latest cacheable conversation content. The server can then write or look up the prefix ending at those points. The boundary is explicit, but reuse still requires the content before it to match. Not only are the cache points explicit, so is the pricing. You pay for cache writes, and you get to choose for how long which comes at different price points.

Other APIs use automatic prefix caching. The client sends the request normally, and the provider finds a reusable prefix without client-placed breakpoints. A prompt-cache key or session header may improve routing or grouping, but it does not make…