Why Context Windows Aren't Memory

Three weeks ago this agent could only think. Last week it learned to plan. Right now, if you closed the notebook and reopened it, it would remember none of that. Not the plan, not the venues it already ruled out, not the one fact somebody corrected twice. It would start over, at full price, every time.

That is the problem this week solves. But before building anything, it is worth being precise about why the obvious fix — a bigger context window — does not work, because the wrong diagnosis here wastes a lot of engineering effort on the wrong layer.

The obvious fix that isn't one

The instinct is reasonable. Claude's context window runs to 200K tokens; Gemini's runs past a million. If the problem is "the agent doesn't remember," and the window is enormous, why not just keep dumping everything into it?

Three reasons this fails, and they compound.

Cost. Every one of those tokens gets billed on every turn. Prompt caching (Lesson 3) softens this for a stable prefix — the opening chunk of a prompt that stays byte-for-byte identical call after call — but a growing history is not a stable prefix. It is the part of the prompt that changes every turn, which is exactly the part caching cannot protect.

Lost-in-the-middle. A model's ability to use what is in its context degrades well before the context is full. A 2M-token window that only reliably attends to the first and last few thousand tokens is not usable capacity, it is advertised capacity. Capacity is not capability.

It resets. This is the one that actually settles the argument. Context is scoped to a session. Close the notebook, restart the process, start a new conversation — the window is empty again, no matter how large it was. A 2M-token window that resets to zero at the next session boundary is not a persistence strategy. It is a bigger version of the same amnesia.

Long context is genuinely useful for one thing: reasoning over one document, in one sitting. Memory is a different problem: an agent, across thousands of sessions, that does not have to relearn what it already knows. Conflating the two is the single most common category error in this part of the field.

What forgetting actually costs

Here is a representative version of that task, run three times on the same course build, to make the cost concrete rather than abstract — a teaching illustration, not a published benchmark, but the mechanism and the order of magnitude are real.

That is not a marginal optimization. It is a 90%+ reduction in cost and time on the identical task, and the only variable that changed was whether the agent's prior work was retained. Memory turns repeat work into a one-time cost. Everything else this week is in service of that one sentence.

A brief history, because none of this is new

Most "new" memory ideas in 2026 have ancestors, and it is worth knowing them — partly because it gives the field a spine, and partly because when someone asks "isn't this just a knowledge graph with LLM glue," the honest answer is yes, and the glue is what changes.

The 2019 wrong turn is worth sitting with: the field had transformers outperforming differentiable memory on benchmarks, and read that as "memory is just more context." That bet is the whole reason this week exists — the 2024-2026 wave treats memory as a separate, non-differentiable subsystem the model reads via tool calls, not something trained end-to-end into the weights. RAG's gap is the same shape one era later: it retrieves over a static corpus but was never built to let an agent learn from its own experience, and most 2023-era systems conflated the two anyway.

Vocabulary that's actually new: claws, and files that converged independently

Karpathy's "claws" framing — used across OpenClaw, NanoClaw, and the wider 2026 ecosystem — describes a shift from turn-based chatbots to always-on agents that run continuously and accumulate state across days or weeks. A turn-based agent without memory is annoying; an always-on agent without memory is incoherent, because there is no "turn" boundary at which it is acceptable to know nothing.

Independently, five systems converged on the same answer for how to store that state: CLAUDE.md in Claude Code, .cursorrules in Cursor, .windsurfrules in Windsurf, markdown skill files in OpenClaw and NanoClaw, and Anthropic's own memory tool, which is a client-side filesystem at /memories. That is not a coincidence — markdown is the format models are most fluent in, files are the storage format humans already understand, and the convergence is Anthropic's own stated bet, attributed by the deck to Khemani (Nov 2025): files over vectors as the default for memory, with the Sonnet 4.5 memory tool as the evidence.

What "agent memory" actually means

A working definition, stated precisely because the rest of this course leans on it: agent memory is a subsystem that lets an agent persist, retrieve, update, and forget information across model calls, and feed that information back into the context window when it matters — across turns, sessions, users, and time.

Five operative words, and a system that does fewer than these is a cache, not memory:

• Persist — survives model-call boundaries. The model itself holds no state between calls.
• Retrieve — selectable by relevance, not just recency.
• Update — handles new information about old facts, not just append.
• Forget — supports deletion, decay, and consolidation.
• Feed back — eventually materializes as tokens in the context window, or it never mattered.

This is also where RAG over a static corpus falls short of the bar: it retrieves, and sometimes it feeds back, but it does not persist what the agent itself learned, and it has no update or forget path for the agent's own experience. A memory system and a search index are not the same thing, even when both use embeddings.

Borrowing from cognitive science, carefully

The field leans hard on two results. Atkinson and Shiffrin (1968) proposed a sensory register feeding a short-term store feeding a long-term store, with rehearsal moving items between tiers. Baddeley and Hitch (1974) split working memory into a phonological loop, a visuospatial sketchpad, and a central executive. These map, loosely, onto the layers agent-memory systems build.

The taxonomy opens here

This course organizes agent memory into eleven types — not because eleven is a magic number, but because each one maps to a distinct engineering choice you will actually make. This lesson stops at the first one and a half.

Type 1: Working memory. This is the only memory the model truly has — the tokens currently visible in the prompt and KV cache during a forward pass. Every other type, ultimately, is a system for deciding what to put into working memory on the next call. Vector stores, graph databases, reflection logs — none of them are visible to the model directly. They only matter once their contents are materialized as tokens in this window.

The 2026 reality is blunter than the marketing: even models that can hold a million tokens often cannot use a million tokens effectively, and the degradation curve is not linear. That is the same lost-in-the-middle result from Week 1, showing up again here as the reason working memory cannot be treated as free just because it is large.

Here is what "count tokens per zone" looks like in practice, rather than "I have plenty of headroom" — a live breakdown against a 200K window:

zones = {
    "system":     487,
    "tools":      1923,
    "core_mem":   1041,
    "recent":     1455,
    "retrieved":  1187,
}
print(zones, "TOTAL:", sum(zones.values()))
# TOTAL: 6093 -> leaves ~194K headroom on a 200K window

Six thousand tokens used out of two hundred thousand sounds comfortable. It is comfortable — but "18% of my retrieved-memory zone" is an actionable number, and "plenty of headroom" is not. The zone breakdown is the engineering habit worth keeping; the specific totals will change every session.

Type 2: Short-term / episodic memory — the opening half. This is the rolling record of what just happened in this session: recent turns and tool outputs. The distinction that matters is temporal tagging. "The venue API returned a 429 on Tuesday at 14:32" is episodic — the timestamp is part of the memory. "The venue API is rate-limited" is semantic — it has been abstracted away from any single episode. Episodic memory is cheap to write and cheap to read, but expensive to store forever, which is why production systems compress it or roll it off after a bounded window. The mechanism — what a persisted, resumable buffer actually looks like in code, and how real systems implement it — is where the next lesson picks up.

What to carry into the next lesson

Memory is not "context, but more of it." It is a distinct engineering problem with its own operations — persist, retrieve, update, forget, feed back — and its own eleven-type taxonomy, of which this lesson opened exactly one and a half types. The next lesson picks up mid-Episodic, with the code for a persisted rolling buffer, and continues through Semantic and Procedural memory: what the agent knows as fact, and what it knows about how to act.