Context & Memory Engineering — The Window as a Resource

It's a Thursday afternoon, and Dev is forty minutes into the best pairing session of his life. He's building a payments integration with a coding agent, and they're in a groove. The agent knows the shape of his codebase. It remembers that he uses snake_case in the database and camelCase in the API. It remembers the weird thing about his Stripe webhook that he explained twenty messages ago. It remembers that he hates default exports. Every suggestion lands. It feels less like using a tool and more like working with a colleague who has been on the team for years.

Then he asks one more question — a simple one, a refactor he's asked for a dozen times before — and the agent gives him a stranger's answer. It suggests camelCase in the database. It re-introduces a bug they fixed an hour ago. It asks him what payment provider he's using, as though the last forty minutes never happened. It has, in the most literal sense, forgotten him.

Dev stares at the screen with the specific betrayal you feel when a coworker forgets your name. What happened? Nothing dramatic. The conversation simply got long enough that the oldest messages — including the careful explanation of his webhook quirk and his naming conventions — slid out the back of the model's attention to make room for the newest ones. The model didn't get dumber. It ran out of room. The colleague who knew him so well didn't quit. He just hit the edge of what he could hold in his head at one time, and the team never gave him a notebook.

This chapter is about that notebook. It is about treating the thing every model has — its context window — not as an infinite scratchpad but as what it actually is: a small, expensive, fixed-size desk. Everything the model can "see" at the moment it answers you must fit on that desk. When the desk fills up, something falls off. The discipline of deciding what goes on the desk, what gets filed away, what gets thrown out, and what gets fetched back when needed is one of the highest-leverage skills in the entire agentic era. We are going to call it context engineering, and by the end of this chapter you will never look at a long chat the same way again.

The Desk and the Filing Cabinet

Let's anchor the whole chapter in one image, because every technique that follows is a variation on it.

Two facts about that desk make it the central resource of agentic systems.

First, the desk is small and fixed. A model advertised with a "200,000-token window" has a desk that holds roughly 200,000 tokens — about 150,000 words, or a 500-page book. That sounds enormous until you watch a real agent work. The system prompt eats some. The tool definitions eat more. Every file the agent reads, every command's output, every one of its own previous thoughts — all of it stacks onto the desk and stays there. A serious coding session can fill 200,000 tokens in an afternoon. When it fills, you are out of room, full stop.

Second, the desk is expensive, and it is expensive on every single turn. This is the fact beginners miss. The model does not read your latest message in isolation. On every turn, it re-reads the entire desk — the whole conversation so far — to produce the next response. By turn thirty, you might type one line, but the model re-reads the system prompt, the tool definitions, and all twenty-nine prior turns before it can answer. A 100,000-token conversation means the model processes 100,000 tokens of input to answer your one-line follow-up. You are not paying for what you just typed. You are paying rent on the whole desk, over and over, every time anyone speaks.

Hold those two facts together — small and re-read every turn — and the thesis of the chapter falls out on its own. The context window is a scarce resource with a recurring cost. And anything that is scarce and recurringly costly must be budgeted, managed, and accounted for — exactly like memory in a program, cash in a startup, or attention in a meeting. Most people treat context as free and infinite. Builders treat it as a budget.

The Context Budget

So let's actually do the accounting. What is sitting on the desk during a typical agent turn? It is never just "the conversation." It is a stack of distinct line items, and naming them is the first step to controlling them.

THE CONTEXT BUDGET (one turn)
==========================================================

  +----------------------------------------------+
  | SYSTEM PROMPT      persona, rules, format     |  fixed
  +----------------------------------------------+
  | TOOL DEFINITIONS   every tool's schema        |  fixed-ish
  +----------------------------------------------+
  | MEMORY / FACTS     pinned notes, prefs        |  small
  +----------------------------------------------+
  | RETRIEVED CONTEXT  RAG chunks, file contents  |  variable
  +----------------------------------------------+
  | CONVERSATION       every prior turn, in full  |  GROWS
  +----------------------------------------------+
  | CURRENT MESSAGE    what the user just said     |  tiny
  +----------------------------------------------+
       total must fit under the window limit
==========================================================

Five of those six line items are things you put there as the builder. Only the last one comes from the user. That should reframe how you think about your job. You are not a passive recipient of whatever the conversation happens to contain. You are the allocator of a fixed budget, and every token you spend on one line item is a token you cannot spend on another.

Let's make the economics concrete, because the numbers change behavior. Suppose your model charges for input tokens, and you run an agent through a 30-turn coding session. By the end, the conversation holds 80,000 tokens. The naive view says: "I sent 30 messages, no big deal." The real view: the model read a growing prefix on every one of those 30 turns. The total input it processed is not 80,000 — it's the sum of every turn's read, which for a steadily growing conversation is roughly the area under a triangle: on the order of 1.2 million input tokens for that single session. The conversation you can see is 80K. The bill you can't see is fifteen times larger.

The micro-lesson: every token on the desk is paid for repeatedly, so the question is never "can I fit this?" but "is this worth re-reading on every future turn?"

What Fills the Desk — and What Spills Off It

Here's the failure mode Dev hit at the start of the chapter, stated precisely. When a conversation grows past the window limit, something has to give. The most common — and most dangerous — strategy is the simplest one: drop the oldest messages. Keep the most recent turns that fit, silently discard everything older. This is a sliding window, and it has a brutal property: the things you established first are exactly the things that vanish first. The naming conventions, the webhook quirk, the "never use default exports" — all of it lives in turns one through three, set down while you were configuring the work. The messy middle and the recent chatter survive; the founding decisions fall off the back. The conventions you most need kept are precisely the ones a sliding window throws out soonest.

This is precisely why the model "forgot" Dev. His naming conventions and webhook quirk were established at the start, when he was setting up the work. The sliding window kept the recent chatter — the part that felt active — and quietly evicted the foundational decisions that should have been permanent. The model wasn't confused about the present. It had been robbed of the past.

So the real questions of context engineering come into focus. When the desk fills, you have only a few moves, and a good system uses all of them deliberately:

• Keep — the live, high-value content (current task, recent turns).
• Evict — genuinely dead content (resolved tangents, stale file dumps).
• Compress — summarize a long stretch into a short, faithful digest.
• Externalize — write important facts to a store and fetch them back on demand.

The rest of this chapter is those four verbs, in order. The art is knowing which one to apply to which content. Throwing out the user's correction of a bug is a disaster. Throwing out the full 400-line file the agent read, debugged, and already fixed is hygiene. They look the same to a sliding window. They are opposites to a good one.

Compaction: Trading Detail for Room

The single most important technique in long-running agents is compaction, also called summarization or context distillation. When the desk approaches full, instead of bluntly dropping old turns, you summarize them. You replace fifty turns of verbose back-and-forth with a tight paragraph that preserves the decisions and state while discarding the play-by-play.

COMPACTION
==========================================================

  BEFORE (52 turns, ~70,000 tokens):
  +----------------------------------------------+
  | full transcript: every file read, every       |
  | command, every dead end, every "hmm let me   |
  | check", every tool result in full ...        |
  +----------------------------------------------+

                    | summarize the old turns
                    v

  AFTER (~2,000 tokens):
  +----------------------------------------------+
  | SUMMARY OF WORK SO FAR:                       |
  | - Goal: add idempotent Stripe webhook         |
  | - Conventions: snake_case DB, camelCase API,  |
  |   no default exports                          |
  | - Done: created webhooks table, dedupe on     |
  |   event_id, tests passing                     |
  | - Open: handle refund.created event next      |
  +----------------------------------------------+
  | + the last few RAW turns, kept verbatim       |
  +----------------------------------------------+

  35x smaller. The DECISIONS survived; the
  play-by-play did not. Room reclaimed.
==========================================================

Notice what the summary kept: the goal, the conventions (the exact thing Dev lost), the completed state, and the open thread. Notice what it threw away: the forty file reads, the three wrong turns, the tool outputs already acted on. That is the whole craft of compaction in one frame — preserve the decisions, discard the deliberation.

A few hard-won rules make compaction work in practice rather than in theory.

Compact at boundaries, not mid-thought. The worst time to summarize is in the middle of a debugging chain where the model is holding a fragile hypothesis. The best time is at a natural seam — a feature shipped, a test passing, a sub-task closed. Trigger compaction on milestones, or on a token threshold (say, 70% of the window), whichever comes first.

Always keep the last few turns raw. Summaries are great for the distant past but terrible for the immediate present, where exact wording still matters. The standard pattern is a hybrid: a compressed summary of everything old, followed by the last N turns kept verbatim. The model gets the gist of the long history and the precision of the recent moment.

Summarize with the model, then verify the summary. A summary is itself a model output, which means it can hallucinate, drop a constraint, or subtly restate a decision wrong. The dangerous bug is a compaction that silently changes "dedupe on event_id" into "dedupe on customer_id." Treat the summary as code: have the model produce it, then — for high-stakes agents — have a cheap check confirm the critical facts (the conventions, the open task) survived intact. A corrupted summary is worse than no summary, because it's confidently wrong and now it's load-bearing.

Externalized Memory: The Notebook the Agent Keeps

Compaction buys you room inside one session, but it can't help across sessions, and it can't perfectly preserve a critical fact forever. For that you need to get the fact off the desk entirely and into durable storage the agent can read and write. This is external memory — the agent's notebook, its filing cabinet, the thing that turns a goldfish into a colleague.

The key conceptual move is to stop thinking of memory as "the conversation history" and start thinking of it as a store the agent operates on with tools. The agent doesn't just passively accumulate context; it actively decides "this fact matters, I'll write it down" and later "I need that fact, I'll look it up." Memory becomes a verb.

It helps to split agent memory into three kinds, because they want different storage and different rules — and the split borrows from how we remember. Working memory is the live context window itself: volatile, in-session, expensive — the desk. Short-term memory is the running summary of this session's progress, the "what we've done so far" that survives compaction but dies with the session. Long-term memory is the set of facts that persist across sessions — user preferences, project conventions, past decisions, hard-won lessons — the filing cabinet.

Working memory is the context window — we've covered it. Short-term memory is the running summary compaction produces — it lives and dies with the session. Long-term memory is the new and powerful idea: a durable store, outside any single conversation, that the agent reads at the start of a session and writes to at the end.

This is exactly how a well-run coding agent fixes Dev's problem. Instead of relying on his naming conventions surviving forty turns of sliding window, the agent writes them, once, to a persistent memory file — conventions: snake_case DB, camelCase API, no default exports. Every future session, that file is loaded onto the desk first, before any conversation. The convention is no longer a fragile message hoping to outlast eviction. It's a pinned fact that starts every session and survives every compaction, because it lives in the cabinet, not on the desk.

What to Write Down — and What Not To

External memory is powerful, which makes it dangerous. The instinct, once you have a notebook, is to write everything in it. That recreates the cluttered-desk problem one level down: a memory store stuffed with every passing remark becomes its own swamp, and loading it back floods the very context you were trying to protect.

The discipline is a filter. Write down facts that are durable, reusable, and would be expensive to rediscover. Don't write down the transient.

• Write it: user preferences ("prefers terse explanations"), project conventions, architectural decisions and their reasons, hard-won lessons ("the staging DB resets nightly — don't trust its data"), stable facts about the user's goals.
• Don't write it: the contents of a file the agent can just re-read, the play-by-play of a debugging session, anything that's true only for the next five minutes, anything already derivable from the codebase.

Retrieval-as-Memory: The Cabinet You Search

We just said long-term memory is loaded "at the start of a session." But what if your memory store grows huge — thousands of past decisions, months of project history, every lesson the agent ever learned? You can't load all of it onto the desk; that's the cluttered desk again, now industrial-sized. The answer is the one technique from Chapter 13, repurposed: don't load all of memory — retrieve the relevant slice of it on demand.

This is where context engineering and RAG touch — and it's worth being precise about the difference, because conflating them is the most common confusion in this whole area.

The mechanism, mercifully, is identical to RAG and you already know it: chunk the memories, embed them, store the vectors, and when a new session starts, embed the current task and retrieve the few most relevant past memories — then put only those on the desk. Imagine a cabinet holding four thousand memories: the webhook dedupe rule, the user's preference for terse answers, the nightly staging reset, the reasoning behind choosing Postgres over Mongo, and thousands more. A new task arrives — "extend the webhook to handle refunds" — and a similarity search pulls just the two folders that matter (the dedupe rule, the Postgres decision) onto the desk. Four thousand memories in the cabinet, two on the desk. You get the benefit of unlimited long-term memory (the cabinet can be enormous) without the cost of unlimited context (only the relevant folder lands on the desk).

This closes the loop on the four verbs. Keep the live turns, compress the old ones into a session summary, externalize the durable facts into a memory store, and retrieve just the relevant slice of that store when a task needs it. A serious long-running agent does all four, continuously, as a kind of metabolism — constantly deciding what to hold, what to fold into a summary, what to write to the cabinet, and what to pull back out.

Prompt Caching: Paying Rent Once

There is one more piece of the economics, and it's the one that turns context engineering from "save tokens" into "save real money at scale." It addresses the wasteful fact we established earlier: the model re-reads the entire desk on every turn. If the front of that desk — the system prompt, the tool definitions, the loaded memory — is identical turn after turn (and it usually is), why should you pay full price to re-process it every single time?

You shouldn't, and prompt caching is the mechanism that lets you avoid it. The provider caches the processed form of a stable prefix. The first time the model reads your big system prompt and tool definitions, you pay to process them and they're cached. On every subsequent turn that begins with the exact same prefix, the model reuses the cached work — at a steep discount, often 90% off the input price for the cached portion — and only pays full price for the new tokens at the end.

PROMPT CACHING
==========================================================

  Each turn's desk = STABLE PREFIX + new stuff

  +=========================+----------------------+
  |  STABLE PREFIX          |  VARIABLE TAIL       |
  |  - system prompt        |  - new user message  |
  |  - tool definitions     |  - new tool results  |
  |  - pinned memory        |                      |
  +=========================+----------------------+
   ^ identical every turn      ^ changes every turn

  Turn 1: process prefix (full price) -> CACHE it
  Turn 2: reuse cached prefix (~90% off) + tail
  Turn 3: reuse cached prefix (~90% off) + tail
   ...
  The big, stable part is paid for ONCE, then rented
  cheaply. Cache the desk's front; pay full only at
  the back.
==========================================================

The catch — and it's a sharp one — is that caching keys on an exact prefix match. The cache only helps if the front of your context is byte-for-byte identical across turns. Change one token near the top and the cache misses from that point onward, and you pay full price again for everything after the change. This has a direct, non-obvious design consequence that separates people who understand context from people who don't.

The micro-lesson: structure context by what changes, not by what matters — stable prefix you can cache, volatile tail you can't, in that order.

Context Rot: Why More Is Not Better

There's a final, counterintuitive truth that ties context engineering to quality, not just cost — and it's the deepest reason the "just use a bigger window" hope falls short. You might assume that as long as a fact fits on the desk, the model uses it perfectly. It doesn't. Models attend unevenly across their context, and as the desk gets crowded, the model's ability to actually use any given fact degrades. Practitioners call it context rot, and Chapter 13 met its cousin under the name "lost in the middle" — the finding that information buried in the middle of a long context gets far less attention than the same information at the beginning or end.

This is the quiet vindication of everything in this chapter. People who don't understand context reach for the biggest window they can find and pack it full, reasoning that more information can only help. Builders who do understand context reach for the smallest context that contains what's needed, because they know three things at once: a fat context costs more (re-read every turn), it caches worse (more volatile content), and — the kicker — it often answers worse (attention spread thin, key facts lost in the middle). Lean context isn't just cheaper. It's frequently better.

Putting It Together: The Long-Running Agent

Let's assemble the whole machine, because in a real agent these techniques don't operate alone — they run together, continuously, as one system. Picture an agent that's been working with a user for weeks, across dozens of sessions, on a large codebase. Here is its full context metabolism on a single turn.

THE CONTEXT METABOLISM (one turn, mature agent)
==========================================================

  SESSION START
  |
  +-- load LONG-TERM MEMORY (the notebook):
  |     conventions, decisions, lessons      [stable]
  |
  +-- load SYSTEM PROMPT + TOOL DEFS         [stable]
  |     (front of desk -> CACHED)
  |
  v
  EACH TURN
  |
  +-- user message arrives                   [volatile]
  |
  +-- RETRIEVE relevant past memories +
  |     RAG knowledge for THIS task          [variable]
  |
  +-- model reasons over the desk, acts
  |
  +-- if desk > 70% full:
  |     COMPACT old turns into a summary,
  |     keep last few turns raw
  |
  v
  SESSION END
  |
  +-- WRITE durable new facts back to
        LONG-TERM MEMORY (update the notebook)
==========================================================

Read that diagram as a sentence and you have the entire chapter: load the stable notebook and prompt at the front so they cache; on each turn fetch only the relevant memories and knowledge for the task; reason; compact the old turns when the desk fills; and at the end, write the durable lessons back to the notebook for next time. Keep, compress, externalize, retrieve — plus cache the stable parts and never crowd the desk. That loop is what separates an agent that forgets you mid-session from one that feels like it has known you for years.

None of it requires a bigger window. All of it requires managing the window you have. And that is the reframe to carry out of this chapter and into the rest of Part VI: the context window is not a feature you wait for vendors to enlarge. It is a resource you engineer — budgeted, compacted, externalized, retrieved, cached, and above all kept lean. The builders who win the agentic era are not the ones with the biggest desks. They're the ones who keep the right things on the desk, file the rest, and always know where to find what they put away.

Dev, by the way, fixed his problem in about four minutes. He didn't switch models or beg for a larger window. He wrote eight lines into a project memory file — his conventions, his webhook quirk, his hatred of default exports — and told his agent to load it at the start of every session. The colleague who forgot him never forgot him again. Not because he got a better memory. Because Dev finally gave him a notebook.

Chapter endnotes

[1] The "lost in the middle" phenomenon — that models use information at the start and end of long contexts far more reliably than information in the middle — was demonstrated by Nelson F. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023), and is the empirical backbone of the "context rot" intuition in this chapter. It is the single best reason to keep contexts lean rather than full.

[2] Anthropic's guidance on prompt caching (docs.anthropic.com) documents the exact-prefix-match requirement and the ordering rule — stable content first, volatile content last — that makes caching effective. The roughly 90% discount on cached input tokens cited here reflects typical provider pricing for cached reads versus fresh processing; exact figures vary by model and provider.

[3] The distinction between working, short-term, and long-term agent memory mirrors the human cognitive model and has become standard vocabulary in agent frameworks. Chip Huyen's AI Engineering (O'Reilly, 2025), Chapter 8, treats retrieval and memory together and is the clearest practitioner's reference for when context should be held, summarized, or externalized.

[4] Project-memory files (such as CLAUDE.md in Claude Code, and analogous mechanisms in other agentic tools) are the practical embodiment of externalized long-term memory described in this chapter: a durable, version-controlled store of conventions and decisions that the agent loads at session start and can write back to at session end.

[5] Compaction (also called context distillation or summarization) is increasingly handled automatically by agent harnesses when a conversation approaches the window limit. The hybrid pattern recommended here — a generated summary of old turns plus the last few turns kept verbatim — is the dominant production approach because it preserves both the gist of long history and the precision of the immediate moment.

[6] The economic observation that input cost grows with the cumulative re-read of the conversation — roughly the area under a triangle for a steadily growing chat — is why context discipline and prompt caching together, not model choice alone, dominate the unit economics of long-running agents at scale.