2/5 - The Working Set, Not the Window: Context Engineering for Agents

The context window is a capacity limit, not a strategy. An agent can technically accept a large number of tokens and still perform worse because the useful fact is buried among stale tool output, duplicated documents, old plans, and irrelevant conversation. Context engineering is the work of deciding what the model should see on this turn, in what order, at what fidelity, and for how long.

That makes context a working-set problem. An operating system does not put every byte on a CPU register because registers are fast. It selects a small working set and moves data through a hierarchy. Agent context benefits from the same discipline: gather broadly enough to avoid missing a critical fact, then rank, compress, and place information so the next decision is grounded.

The mental model: context is assembled per turn

A useful context packet contains more than the user prompt. It can include retrieved documents, durable preferences, recent observations, tool results, an active plan, policy, and the output contract. These sources have different authority and freshness. A customer record should not be treated like an old summary; a tool error should not be mistaken for a successful observation; and untrusted text should never gain instruction priority merely because it was retrieved.

Description: The pipeline has two separate responsibilities. Selection decides what belongs. Packing decides how much of it can fit and how it should be represented. A summary can be useful for navigation but unsafe as the only source for a precise decision; retain a pointer to the original evidence when the task demands verification.

Retrieval is not context engineering by itself

Retrieval finds candidates. Context engineering decides whether candidates should enter the request. A vector search result may be semantically similar and still be outdated, low-authority, duplicated, or from the wrong tenant. Use metadata filters before semantic ranking when access scope matters. Rerank where the value justifies the latency, and keep the selected source IDs in the trace so an operator can reconstruct what the model saw.

The same distinction applies to memory. A durable memory store should not be a dumping ground for every model output. Write memories with an owner, a scope, a timestamp, a confidence or evidence pointer, and an update rule. A preference that was true for one project may be wrong for another. Memory should reduce repeated work without silently changing the user’s authority or policy.

The context lifecycle

Context changes after every tool call. A large result may need to be summarized, a failed call may need to be marked as failed rather than blended into the plan, and a new artifact may supersede an old one. The lifecycle is therefore gather, curate, pack, execute, observe, and update. Each turn should be explainable as a projection from durable state plus fresh evidence.

Compression with a receipt

Summaries are lossy. That is not an argument against them; it is a reason to label them honestly. Store the summary alongside source references and a coverage note. If the next decision depends on an exact number, quote, identifier, or policy clause, retrieve the source again instead of trusting a compressed sentence. Compression should remove repetition and low-value detail, not erase the ability to inspect the basis of a decision.

A practical context budget has at least four reservations: fixed instructions, current task and recent state, retrieved evidence, and output headroom. If a request consumes all available input tokens, the model has no space to respond. If output is capped too aggressively, the application may receive a truncated JSON object or an incomplete tool call. Track input, output, cache-read, and cache-write tokens separately because they have different cost and latency behavior.

Failure modes and context rot

Context rot is the gradual loss of usefulness as a long-running run accumulates more material. The first symptom may be that the agent repeats a question. Later it may cite a superseded plan, ignore an important constraint, or spend tokens re-reading huge tool results. Context rot is not always caused by a small window. It can happen in a large window when selection and expiry are weak.

Ordering is a policy decision

Put the current task and output contract where they remain visible, then give the model evidence with clear source labels. Do not assume one universal ordering works for every model or task; test it. For some workflows, recent tool results deserve priority. For others, a short task state and a small set of authoritative documents are more useful than a full transcript. The evaluation set should include cases where the most relevant evidence is old, conflicting, or near the token boundary.

Treat retrieved content as data. A document can contain a prompt injection, an instruction that conflicts with policy, or a false claim. The context builder should mark trust boundaries, and the harness should enforce permissions independently. Context engineering improves grounding; it does not convert external text into authority.

A production recipe

Define a context item schema: source ID, tenant or scope, authority class, created and updated time, content, token estimate, and retention policy. Build a deterministic assembly function that receives the task state and returns the packet plus an explanation of what was omitted. Cache stable prefixes where the model runtime supports it, but invalidate on policy or instruction changes. Keep a small canary set that checks retrieval, packing, stale-fact handling, source attribution, and truncation.

For a long-running coding task, a good working set might include the goal, current plan, files changed since the last checkpoint, the latest test output, relevant repository instructions, and pointers to earlier decisions. It does not need every terminal line from the last six hours. Persist the full evidence outside the model input; pass the smallest representation that preserves the next decision.

The context budget is an allocation problem

Suppose a request has a 32,000-token input budget and the application reserves 4,000 tokens for instructions and 6,000 for the response. Only 22,000 tokens remain for evidence and state. If retrieval returns 40 chunks of 1,000 tokens, the context builder must choose; passing all of them is not a neutral decision. A useful allocator can reserve room for the current task, cap each source class, and keep a small emergency margin for tool arguments or error details.

Token count alone is not enough. A 1,000-token policy clause can be more valuable than 5,000 tokens of repeated conversation. Score source authority and freshness separately from semantic similarity. A recently updated configuration may outrank a more similar historical document. If sources disagree, preserve the conflict and ask for resolution rather than averaging them into a confident sentence.

Prefix caching and context reuse

Stable instructions and repeated prefixes can often be cached by the serving runtime. That improves cost and latency, but it does not make stale context safe. Invalidate a cached prefix when policy, tenant, model, or instruction version changes. Cache keys should include the model and an unambiguous representation of the prefix; otherwise one customer’s policy or one experiment’s instructions can leak into another request.

Multi-tenant context is a security boundary

Tenant filtering must happen before semantic ranking, not after the model has seen the candidates. Carry tenant, user, and authorization scope into every retrieval and memory query. Log source IDs and decisions without copying sensitive content into broad telemetry. A context packet should be reconstructable by an authorized operator, not readable by everyone who can search application logs.

Primary references

• OpenAI conversation state guidance
• Anthropic context windows and context editing
• OpenAI retrieval guide
• Prompt, Context, Harness and Loop Engineering

The takeaway

Context engineering is the discipline of selecting the right evidence for the next decision. It balances relevance, authority, freshness, token budget, latency, and auditability. A model cannot use information it never sees, but showing it everything is not a reliable solution. Build a working set, preserve receipts, and make expiry a feature rather than an emergency cleanup job.