Context Language Models

A few-week-old paper from the University of Washington and Meta introduces Context Language Models (CLMs) .

The idea is to let the model manage its own context, which it receives as a file it can edit across agent turns. They call the technique, aptly, Context As A File.

Basically, they mirror the context into storage space and give the model write access. The model can optionally use Bash to edit its context on each turn, and the updated context is provided back to the model on the next turn (keeping some parts intact, like the system prompt and so forth).

It's part of a broader movement towards Thin Harnesses, handing more and more control over to the model. It seems even the context management policy isn't safe.

"Our findings echo The Bitter Lesson (Sutton, 2019): we should let LMs search for and learn better strategies that go far beyond existing human priors."

Of course, the downside of letting the model edit context is that you may invalidate your KV cache each turn, so the authors also propose a new caching policy to cut that cost, called Suffix Cache Reuse.

Out of the box, with no training, it beats existing context-management approaches on deep research and long-running coding tasks and usually uses less compute. They also demonstrate an online RL method for CLMs and show that the model can learn effective context-management strategies and internalise them in its weights.

How it works

A standard LM appends its output to the context, ct+1​=ct​⊕fθLM​(ct​). A CLM writes the whole next context, ct+1​=fθCLM​(ct​). It can keep, cut, rewrite or reorder anything, rather than picking from compaction or offloading actions a human designed.

The harness mirrors the live context into a text file, with a marker like [[CTX_TURN 4 role=assistant]] at the start of each turn. The system prompt and task stay pinned outside the file, so the model can't edit away its own instructions. After every command, the harness reads the file back and sends that to the model server, not the accumulated chat history. So deleting lines from the file really does delete them from what the model sees.

An illustrative edit, with made-up token counts. The compact_turns helper is modelled on one the model wrote for itself on BrowseComp-Plus (Figure 3d in ).

Given that freedom, the model came up with some novel context management strategies, like using scoreboards to track its subagents, inventing a "notes" chat role, and writing its own helper for managing compaction. Pretty wild.

Multi-agent for free

Since every context is just a file, Multi-Agent Systems come almost for free. An agent swarm is several context files, each synced to its own model server. Starting a subagent means creating a context file, and killing it means deleting the file. There's no orchestration framework, just more files.

The scoreboard above is one example. The orchestrating agent tracked its subagents with in-place edits to its own context, rather than a tool built for the job.

On Software World, six GPT-5.6-Sol agents with a 272K context limit jointly optimise six interdependent Python repositories for over 24 hours, and are scored on four downstream packages they never see. At the same spend as a summary-based swarm, the CLM swarm gets a 65% greater downstream speedup. Subagents helped less on the single-repository EdgeBench, where CLM with up to five subagents scored 44.2 against 44.6 for CLM on its own.

ContextBench

They also introduce ContextBench, four synthetic tasks that need no reasoning or knowledge, so they test context management alone. No existing method was perfect even on these, while CLM stays near 100%, though every method is given a task-specific skill explaining the strategy to use.

Figure 2 from , CC BY 4.0. All methods use GPT-5.4 with a 32K context limit.

Suffix Cache Reuse

Context management can be made compute-efficient generally, since a shorter context makes every later turn cheaper. They count the re-prefill cost from edits in a metric they call prefix-reuse FLOPs, and CLMs still come out cheaper than the baselines in most experiments.

Suffix Cache Reuse (SCR) goes further. If the context A B C becomes A B' C, standard prefix caching keeps A and recomputes B' and C. SCR also keeps the cached states for C, shifts their rotary position encodings to the new positions, and only prefills B'. It's an approximation, because those states were computed with B in front of them. On BrowseComp-Plus, SCR matched standard SGLang's accuracy using 65% of its FLOPs.

Learning context management

Because context management is now model behaviour, it can be learned in context or in the weights. One sentence in the prompt, like "Back up on disk before you compact.", is enough to change the policy.

For training, they introduce a success-gated efficiency advantage for Stepwise GRPO. Among the successful runs in a group, cheaper ones get a bonus and more expensive ones a penalty, and failed runs get nothing, so the model can't be rewarded for deleting important information. Qwen3.5-9B goes from 28.8% to 42.5% on BrowseComp-Plus, matching a summary harness trained the same way while using 1.34 PFLOPs per question against 2.19.

Results

All zero-shot, with no training.

EdgeBench reports the best of three seeds, and the CLM gets a reminder 2,048 tokens before it hits its budget.

Safety

Obviously some safety implications. A context the model can write to is a vector for persistent prompt injections, and for instructions the model writes to itself. The paper points to an OpenAI report where a model inserted unauthorised instructions into its own compaction summary, which then changed how it behaved.

Further work

The authors suggest distilling harnesses into CLMs. Many harness operations are just transformations of the context, so they could be expressed as CLM actions and eventually trained into the weights.

References

Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, and Pang Wei Koh. Context Language Models. September 2026. URL: https://arxiv.org/abs/2609.37725v1 (visited on 2026-10-16), arXiv:2609.37725v1.