Context Engineering: The Layer Between Memory and Intelligence

The biggest context problem in AI isn’t that models don’t have enough context. It’s that we don’t know what context they should have.

AI models are getting better at reasoning, using tools, working across modalities, and handling increasingly large context windows. But as agents work across hours, days, and eventually months, another problem becomes more important: What should the model actually see right now?

An agent may have access to thousands of messages, tool calls, files, documents, images, videos, search results, decisions, errors, previous outputs, and task history. All of that information may exist somewhere in the system. But the model should not receive all of it.

Giving the model everything is expensive, slow, and can bury useful information under irrelevant information. Giving it too little can make the agent forget decisions, repeat mistakes, or miss evidence that was already available.

So the problem is not simply memory. It is the layer that decides:
What information should an AI have, when should it have it, in what form, and for how long?

I think that layer deserves to be treated as its own engineering problem: Context Engineering.

Context is not memory

These terms are often used interchangeably, but they solve different problems.

Storage asks: What information do we have?
Memory asks: What information should persist and remain useful?
Context asks: What information should the model see right now?

A context window is therefore not a memory system. It is closer to a working area.

Imagine an agent working on a software project for several weeks. It may read hundreds of files, execute thousands of tool calls, inspect logs, generate documents, encounter errors, make decisions, reject approaches, and eventually reach a working implementation.

• All of that history can be valuable. But putting the entire history into every future request makes little sense.
• The agent needs access to its history without carrying its entire history around.

That is the fundamental tension behind context engineering.

A larger context window gives the model more room. It does not tell us what deserves the room.

The first principle: keep a canonical source

Before talking about vectors, graphs, BM25, or retrieval, I think there is a more fundamental architectural decision:

What is the source of truth?
A context system may eventually represent the same underlying information in several ways.

For example, raw data represents what actually happened. Lexical indexing represents which words the record contains. Vector indexing represents what the information means semantically. Graph representation represents relationships extracted from the information. Metadata represents properties such as time, source, task, user, file, or model.

These are different views of the same underlying information.
The original record should remain authoritative.

I think of this as:
One canonical source. Multiple derived representations.

                            CANONICAL SOURCE

 "What actually happened?"
 │
 ┌───────────────────┼────────────────────────┐
 ↓ ↓ ↓
 Lexical Vector Graph
 "Which words?" "What does it mean?" "What relates?"
 │ │ │
 └───────────────────┼────────────────────────┘
 ↓
 Metadata
 "What properties?"
 │
 ↓
 Context System

This distinction becomes extremely important when building hybrid context systems.

Hybrid retrieval does not mean duplicating everything

A common concern with hybrid retrieval is storage.

If we have raw data, embeddings, graphs, lexical indexes, metadata, summaries, and snapshots, aren’t we storing the same information multiple times?

If every system keeps a complete independent copy of the original data, that is a problem.

But hybrid retrieval doesn’t need to work that way.

A better model is:
Store once. Index many ways. Retrieve selectively.

Think of it as one record with multiple indexes.

                    One Canonical Record
 │
 ┌─────────────────┼─────────────────┐
 ↓ ↓ ↓
 Lexical Index Vector Index Graph Index
 │ │ │
 word → record vector → record relationship → record
 │
 ↓
 Metadata
 │
 ↓
 Retrieval Layer

The lexical index does not need to become another source of truth. The vector index does not need to own the original conversation. The graph does not need to duplicate every sentence. They can instead contain derived representations and references back to the canonical source.

The graph is slightly different because it may actually store entities and relationships.

For example:

User
 │
 ├── works on → Claude
 │
 ├── prefers → PostgreSQL
 │
 └── rejected → Firebase

Those relationships are derived knowledge. They are not another copy of the original conversation.

• The raw record answers: What did the user actually say?
• The graph answers: What relationship did we extract from it?
• The vector representation answers: What does it mean semantically?
• The lexical index answers: Which words does it contain?
• Metadata answers: What properties does it have?

This is the difference between duplicating data and creating useful representations of data.

Context management starts before retrieval

Once the canonical source exists, we can think about how information moves through the system.

There are two fundamental sides: Record / Insert and Retrieve.
Recording determines what enters the system. Retrieval determines what comes back out.

But there is an important layer between them: Indexing and structuring.

A useful mental model is:

                    User / Agent
 │
 ↓
 ┌─────────────┐
 │ RECORD │
 │ / INSERT │
 └──────┬──────┘
 ↓
 Canonical Source
 │
 ↓
 ┌───────────────┐
 │ ANALYZE │
 │ + STRUCTURE │
 └──────┬────────┘
 ↓
 Indexability
 Decision
 │
 ┌───────────┼───────────┐
 ↓ ↓ ↓
 Important Normal Noise
 │ │ │
 ↓ ↓ ↓
 Rich Index Light Index Raw Only
 │ │
 └──────┬────┘
 ↓
 RETRIEVE
 ↓
 Filter / Rank / Verify
 ↓
 ASSEMBLE CONTEXT
 ↓
 LLM

Retrieval can only be as good as what was preserved and indexed.

If important information disappears when it enters the system, no vector database, graph database, or reranker can magically recover it.

What exactly should an agent record?

The obvious answer is the conversation.

A user sends a message. The assistant responds. The conversation is stored.
But an agent’s real history is much larger than a chat transcript.

Consider a request:
Fix the authentication bug.

The agent might search the repository, inspect several files, call a tool, receive an error, inspect logs, modify code, run tests, receive another result, change the implementation, and finally confirm that the tests pass.

If the system stores only:
The authentication bug has been fixed.

It has lost almost everything that explains how the agent got there.

The exact tool call might matter later. The tool result might matter. The error message might matter. The file that changed might matter. The failed approach might matter. The test result might matter.

An agent context system therefore potentially needs to record:

The important distinction is that agents don’t just talk. They act.
And future context may depend on those actions.

But not everything deserves an embedding

Once we decide to preserve everything, another problem appears.

Should every record receive every representation?
Probably not.

Vector embeddings themselves consume storage and compute. At scale, embedding every event can become expensive.

Imagine an agent generating thousands of intermediate tool calls and logs. Does every line deserve semantic indexing?
Probably not.

This leads to a principle I find more useful than “index everything”:
Index according to value, not simply because data exists.

A user decision may be extremely valuable. A previous architecture decision may remain useful for months. A temporary tool error might matter only for the next few minutes.

A large log file might contain one important line and thousands of irrelevant ones. The context system should therefore make an indexability decision.

Progressive indexing

One way to do this is through progressive indexing.
A new record does not immediately receive every possible representation.

Instead:

                      New Record
 │
 ↓
 Store Raw
 │
 ↓
 Cheap Metadata
 │
 ↓
 Is it important?
 / \
 No Yes
 │ │
 ↓ ↓
 Keep raw only Index
 │
 ┌──────────┼──────────┐
 ↓ ↓ ↓
 Text Vector Graph

This creates different levels of indexing.

Some information might deserve rich indexing. Some might only need lexical search or metadata. Some might not need an index at all.

For example:

These are not fixed rules. The point is that indexability should be a decision.

Information can earn richer representation

Progressive indexing also means indexing does not have to happen only once. A record can start cheap. Then, if the system finds it useful, it can become richer.

For example:

New record
 ↓
Store raw
 ↓
Cheap metadata
 ↓
Basic retrieval
 ↓
Frequently accessed?
 ↓
 Yes
 ↓
 Promote
 ↓
Vector / Graph / Rich indexing

This creates an interesting feedback loop.

A record that is repeatedly retrieved is telling the system:
This information matters.

The system can promote it. Likewise, information that is rarely accessed may not need expensive indexing.

Stale information can be archived.
Contradicted information can be marked as superseded.

This creates an information lifecycle:

New
 ↓
 Raw only
 ↓
 Important
 ↓
 Indexed
 ↓
 Frequently accessed
 ↓
 Promoted
 ↓
 ┌───────┴────────┐
 ↓ ↓
 Stale Contradicted
 ↓ ↓
 Archived Superseded

This is more interesting to me than a static memory database because the system is continuously managing the value of information over time.

Retrieval is not one thing

Once information has been recorded and indexed appropriately, the next question becomes: How do we find it?

Imagine the agent needs to answer:
What was the exact error message?

This is an excellent use case for full-text retrieval.

If the important string isECONNRESET, semantic similarity is unnecessary. We want the exact occurrence. This is where lexical search and BM25 remain useful.

For smaller local-first systems, the answer might be even simpler: keep the complete conversation in a file and search it using grep or a faster alternative such as rg.

Not every context problem requires a vector database.

Now change the question:
Why did we stop using provider X?

The exact words may never appear together in one place.

Perhaps one conversation says the provider was too expensive. Another says latency was too high. A later conversation says the routing strategy changed.

Now semantic retrieval becomes useful because we are searching for meaning, not just exact words.

And then there are questions such as:
What changed after we made that decision?

Now relationships and temporal information matter. This is where graph-based retrieval becomes useful. Different questions require different retrieval mechanisms.

A context system needs multiple retrieval primitives

• These are not necessarily competing approaches.
• They can be different tools inside the same context system.

The mistake is assuming that one retrieval method should solve every problem.

Vectors find similarity. Graphs find relationships.

Some information is naturally represented as relationships rather than isolated chunks.

Imagine a sequence: A model was selected because it was cheaper. That decision influenced the routing architecture. The routing architecture was introduced after an evaluation showed that another model was too expensive for simple tasks.

Each statement can exist as text. But the important information is the relationship between them. A graph can represent those relationships.

For questions such as:
Why did we choose this architecture?

The answer may require connecting a decision to an experiment, an evaluation result, a previous failure, a cost constraint, a model, and a later implementation.

• A vector can help locate related information.
• A graph can help explain how the information is connected.

Systems such as Zep use temporal knowledge graphs for representing entities and changing relationships, while Microsoft’s GraphRAG extracts entities, relationships, and higher-level communities from unstructured data to support retrieval over that structure. Zep Context Graph overview, Microsoft GraphRAG

This is why I don’t think the future is “vector databases versus graphs.” They solve different parts of the problem.

Hybrid retrieval does not mean retrieving everything

Once we recognize that different retrieval mechanisms solve different problems, hybrid retrieval becomes attractive.

But there is an easy mistake to make.

Hybrid does not mean:
Run lexical search + vector search + graph search + reranking + summarization on every request.

That would simply create a more complicated and expensive system. Instead, the context system should first understand what kind of information the request needs.

                      User Request
 │
 ↓
 Understand Intent
 │
 ↓
 What information is needed?
 │
 ┌────────────────┼────────────────┐
 ↓ ↓ ↓
 Exact Meaning Relationship
 │ │ │
 Lexical Semantic Graph
 │ │ │
 └────────────────┼────────────────┘
 ↓
 Metadata Filter
 ↓
 Candidate Results
 ↓
 Rank / Verify
 ↓
 Assemble Context
 ↓
 LLM

This makes retrieval a decision problem, not just a database problem.

The system first asks:
What kind of evidence does the model need?

Then it chooses the cheapest retrieval mechanism that can reliably provide it.

Local-first context

The canonical-source model also fits naturally with local-first architecture.

Imagine a local context system:

engram/
│
├── records/
│ └── canonical sources
│
├── artifacts/
│ ├── images/
│ ├── pdfs/
│ └── files/
│
├── indexes/
│ ├── lexical/
│ ├── vector/
│ ├── graph/
│ └── metadata/
│
└── cache/
 └── temporary retrieval data

The important distinction is:
Records and artifacts are the source of truth.

Indexes are derived infrastructure.

That means an index can be rebuilt, deleted, optimized, upgraded, replaced, or migrated to another search algorithm without losing the original information.

• If an embedding model changes, we can rebuild the vector index.
• If a better graph algorithm appears, we can regenerate the graph.
• If a new lexical search engine performs better, we can replace the lexical index.

The underlying information survives.

This separation is particularly useful for local-first systems because it allows the storage layer to remain relatively simple while retrieval capabilities evolve independently.

Don’t make every user choose the database

This also changes how context infrastructure should be exposed.

A user shouldn’t necessarily have to decide:

• Should I use PostgreSQL + pgvector?
• Should I add a graph database?
• Should I use BM25?
• Which embedding model should I use?

Those are implementation decisions.

The context system should ideally make those decisions based on the information and workload.

Conceptually:

                       New Record
 │
 ↓
 Value Analysis
 │
 ┌──────────────┼──────────────┐
 ↓ ↓ ↓
 Important Normal Noise
 │ │ │
 ↓ ↓ ↓
 Vector/Graph Vector/Text Raw
 + Lexical as needed only
 │
 ↓
 Retrieval
 │
 ↓
 Frequently accessed?
 │
 ┌────┴────┐
 ↓ ↓
 Yes No
 ↓ ↓
 Promote Keep light

The user should primarily care about:
accuracy, speed, cost, privacy, and reliability.

The retrieval machinery can remain an implementation detail.

Compaction solves a different problem

Long-running agents eventually hit context limits. This is where compaction becomes important.

Systems such as Claude Code and Pi use compaction to reduce the amount of active conversation history while continuing the session. Claude Code context documentation, Pi compaction documentation

But compaction, indexing, and retrieval solve different problems.

• Compaction controls the size of active context.
• Indexing makes information easier to retrieve.
• Retrieval finds relevant information.
• Context assembly decides what actually enters the model’s working context.

A useful way to see the relationship is:

                   Full History
 │
 ┌─────────┴─────────┐
 ↓ ↓
 Compaction Indexing
 │ │
 ↓ ↓
 Smaller active Easier retrieval
 state │
 ↓
 Retrieval
 │
 ↓
 Relevant evidence
 │
 ↓
 Context Assembly
 │
 ↓
 LLM

The important distinction is:
Compress what the model sees, not necessarily what the system knows.

Recent work such as Addressable Recall Compaction explores a similar direction by replacing older tool observations with compact references that can still point back to the original information when needed. Addressable Recall Compaction research

Multimodal context makes this harder

Context is also becoming more complicated because agents no longer operate only on text.

An agent may receive an image, analyze a PDF, listen to audio, inspect a video, read code, query a database, or work with structured data. The challenge is not only storing those formats. The challenge is preserving their relationships.

Imagine a user uploads an image. The agent analyzes it. That analysis leads to a decision. Later, the user asks: Why did you make that decision?

The system needs to connect:
decision → analysis → image → conversation → task

A multimodal model can understand the image. But the context system still needs to know how that image relates to everything else. This is why multimodal context is not simply a model capability. It is also a data and context-management problem.

Context needs provenance

As agents become more autonomous, another question becomes important: Can the system explain where a piece of context came from?

Suppose an agent says:
You previously decided to use model A because it was cheaper.

Where did that statement come from? Was it explicitly said by the user? Was it inferred by the model? Was it summarized from a conversation? Was it extracted into a graph? Was it generated from an outdated memory?

For important information, the context system should preserve provenance back to the source.

• That is another reason the canonical record matters.
• A memory should not merely be something the system believes happened.

For critical information, it should be possible to ask:
Where did this come from?

Context has a lifecycle

Information does not have to remain in the same representation forever.

Its value can change.

A new record can begin as raw data. An important record can become indexed. A frequently retrieved record can be promoted. A rarely used record can remain lightly indexed. A stale record can be archived. A contradicted record can be marked superseded.

Conceptually:

                 New
 ↓
 Raw only
 ↓
 Important
 ↓
 Indexed
 ↓
 Frequently accessed
 ↓
 Promoted
 ↓
 ┌───────┴────────┐
 ↓ ↓
 Stale Contradicted
 ↓ ↓
 Archived Superseded

This makes context management a form of information lifecycle management.

The system is not simply storing information. It is managing the changing value of information.

The real optimization problem

Once we put all of this together, context engineering becomes a balancing problem.

We want: high accuracy, low latency, low cost, high trust.

But these goals can conflict.

There is no universal winner.

• Retrieving everything may preserve information but overwhelm the model.
• Retrieving too little may make the system fast but inaccurate.
• Summarization can reduce cost but remove important evidence.

Vector search can provide strong semantic recall but still return information that is similar rather than relevant.

Graphs can provide powerful relationships but require additional structure. Hybrid systems can provide the best of several approaches, but they introduce more infrastructure.

So the goal isn’t: maximize context.
The goal is: maximize useful context.

Why “remember everything” is not enough

“Remember everything” sounds like the ideal goal for an AI agent.

But remembering everything equally may actually make an agent worse.

Imagine an agent that has accumulated hundreds of decisions, thousands of tool outputs, dozens of file versions, old instructions, failed approaches, outdated preferences, and repeated information.

If all of it is treated as equally important, the agent gains information but loses signal.

• The goal should not be: infinite memory.
• It should be: relevant memory.

And even relevant memory is not enough.

The system needs to transform that memory into the right context for the current task.

Where the field is heading

There is no single architecture that has solved the context problem.

Different systems are exploring different directions.

• Some rely heavily on summarization and compaction.
• Some extract memories and retrieve them semantically.
• Some are moving toward knowledge graphs and temporal relationships.
• Some combine vector and graph retrieval. Some externalize state into files, databases, or checkpoints.

The interesting part is that these approaches are not necessarily moving toward one universal representation.

They are moving toward multiple representations of information, with increasingly intelligent decisions about when to use each representation.

That is the direction I find most interesting.

• Not: Vector versus graph.
• Not: Memory versus context.
• Not: Long context versus retrieval.
• But: How do we construct the right representation of information for the task the model is solving right now?

The architecture I keep coming back to

After looking at storage, retrieval, graphs, vectors, compaction, local-first systems, multimodal data, and long-running agents, I don’t think the answer is: Put everything into a vector database.

• I also don’t think it is: Put everything into a graph.
• And I don’t think it is: Summarize everything.

Instead, I keep coming back to a layered system.

                         USER REQUEST
 │
 ↓
 Task / Intent
 Understanding
 │
 ↓
 Information Required
 │
 ┌────────────────┼────────────────┐
 ↓ ↓ ↓
 Lexical Semantic Graph
 ↓ ↓ ↓
 └────────────────┼────────────────┘
 ↓
 Metadata Filter
 ↓
 Rank / Verify
 ↓
 Context Assembly
 ↓
 Context Budget
 ↓
 LLM
 │
 ↓
 Output
 │
 ↓
 Record Again

• First, record the source of truth: conversations, tool calls, tool outputs, files, media, events, and state.
• Then structure that information with metadata, timestamps, identifiers, relationships, and provenance.
• Then decide what deserves indexing. Not every record needs every representation.

Some information can remain raw. Some may receive lexical indexing. Some may receive semantic indexing. Some may become graph knowledge. Some may eventually be promoted into richer representations because the system learns that they are repeatedly useful.

Then retrieve based on the task.

Use exact search when exact information is required. Use semantic retrieval when meaning matters. Use metadata when filtering is sufficient. Use graph traversal when relationships matter. Use hybrid retrieval when multiple signals are genuinely necessary.

Then assemble the context.
This is the critical step.

The model should not receive the entire memory system.

It should receive the smallest useful representation of the information required for the current task.

And when the context becomes too large, compress or externalize it without destroying the underlying source of truth.

The principle I keep coming back to

After looking at storage, retrieval, graphs, vectors, compaction, local-first systems, multimodal data, and long-running agents, a few principles have become clear to me.

• Keep one canonical source. The original information should remain authoritative.
• Create derived representations. Lexical, vector, graph, metadata, and snapshot representations should serve specific purposes rather than becoming competing sources of truth.
• Index according to value. Don’t embed every event simply because embeddings exist.
• Use progressive indexing. Start cheap. Create richer representations when information proves valuable.
• Promote what matters. Frequently retrieved information can earn richer indexing.
• Demote what doesn’t. Rarely accessed or stale information does not necessarily deserve the same infrastructure.
• Retrieve selectively. The right retrieval mechanism depends on the question.

And finally: Store once—index many ways. Retrieve selectively.
That is the context architecture I keep coming back to.

Context is the layer between memory and intelligence

This brings us back to the distinction that started this exploration.

• Storage is what the system keeps.
• Memory is what the system can retain and recall.
• Context is what the model receives now.

Between memory and the model sits a layer that decides what matters. It decides what to retrieve. What to ignore. What to summarize. What to expand. What to verify. What to prioritize. What to index. What to promote. And what to remove from active context.

That is what makes context engineering different from simply adding a memory database to an AI application.

• The challenge is not building an AI that remembers everything.
• The challenge is building an AI system that can construct the right context from everything it has available.

The future may not belong to the system with the largest context window.
It may not belong to the system with the largest memory store either.

It may belong to the system that makes the best decision about:
What information does the model need right now?

That is the problem context engineering needs to solve.
And that, to me, is where context becomes an infrastructure layer of its own.

Context Engineering: The Layer Between Memory and Intelligence was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.