The Lost-in-the-Middle Problem

Lesson 01 left you with a demo that half-worked: a small model recovered a hidden needle from the start, middle, and end of a repetitive filler text, seemingly undermining the whole lost-in-the-middle story. That wasn't a mistake in the demo. It was the boundary condition. This lesson runs the harder version of the same experiment and shows you exactly where the phenomenon actually bites — plus the mechanism that makes it inevitable, and the two fixes that work.

The "8K rule," done right

The "8K rule" gets taught badly, almost everywhere. The common version: performance falls off a cliff at 8,000 tokens. The lesson's own framing corrects this to something more precise and, honestly, more useful — it's a curve, not a cliff. Performance degrades gradually as relevant information moves toward the middle of a long context, and roughly 8–10K tokens (about ten pages) is simply the point where that degradation becomes statistically measurable in controlled tests. That reframing matters because it changes what "fixing" the problem looks like. If the 8K figure were a hard wall, the fix would be "stay under 8K tokens." Since it's a gradual curve driven by how distinguishable your signal is from the noise around it, the fix is a data-formatting problem — how you present information to the model — not a token-counting problem.

The lab: mapping the U-curve on real prose

Lesson 01's demo used repetitive, synthetic filler, which is exactly why the needle survived at every position — uniform noise is trivial for attention to filter out. This lab replaces that filler with real, semantically varied prose: the full text of The Great Gatsby, fetched directly from Project Gutenberg (gutenberg.org/cache/epub/64317/pg64317.txt). The needle is a distinctive marker string, SECRET_PUB_CODE=HAYMARKET99, embedded at the start, middle, or end of a window built from that corpus, tested against the same small model, google/gemma-2-2b-it.

python gatsby_ucurve_lab.py

At MAX_CHARS=31_128, the deck's own stated typical result is: the needle is found at the middle and the end, but missed at the start. That's worth pausing on, because it is not the classic U-curve (found at both edges, lost in the middle) — it's what this lesson's own diagnostic table calls a capacity failure: the model's working memory is too shallow to anchor to a prefix this long, so primacy collapses before the middle does. Swap in a larger model, meta-llama/Llama-3.1-8B-Instruct, and rerun at growing window sizes (8,000 → 16,000 → 32,000 characters), and primacy recovers — at which point the true U-curve emerges as you scale up, with the middle degrading first. Small model, capacity problem. Larger model at the same task, the real signal-dilution problem. Different failure modes, same underlying cause, different point at which it shows up.

The mechanism: why this isn't a bug

Here's the part that makes this an engineering fact rather than a training accident to be patched away. Every attention head in a Transformer scores the query token against every other token, and those scores pass through softmax, which forces them to sum to exactly 1.0. That's a fixed budget. At 10 tokens, each relevant token can get roughly 10% of the attention weight. At 1,000 tokens, roughly 0.1%. At 100,000 tokens, roughly 0.001%. As context length L grows, the maximum possible attention weight any single token can receive approaches 1/L — call it vanishing attention: not a separately published finding, just the arithmetic above, spread thinner as the token count grows. More tokens does not mean more memory. It means the same fixed attention budget divided into smaller and smaller slices — a deeper place to bury your own signal, not a bigger place to keep it visible.

This same U-shaped curve has a human precedent. Psychologists documented the Serial Position Effect in human memory in 1913: items at the start of a list move to long-term memory (primacy), items at the end stay in working memory (recency), and the middle is the least reinforced. The parallel to an LLM's context window is direct — the system prompt sits at the top with the highest attention weight, the latest user message sits at the bottom with the second-highest, and everything in between (retrieved documents, tool outputs, conversation history) competes for what's left.

The original empirical version of this finding, cited throughout this lesson, is Liu et al. (Stanford, published at TACL 2023): models given a question and 10–30 documents scored highest when the answer sat at the very start or very end, and dropped substantially when it sat in the middle. The most striking result in that paper is a "negative gain": GPT-3.5's accuracy with the answer buried in the middle of a long prompt fell below its own closed-book baseline — meaning the model performed worse with the correct answer somewhere in its context than with no context at all. Adding information, placed badly, made the model dumber.

It gets worse with scale, not better

The natural assumption is that this is an artifact of smaller or older models, and that frontier-scale context windows in 2026 have mostly solved it. Chroma Research tested that assumption directly in a 2025 study they named Context Rot, running 18 frontier models — GPT-4.1, Claude 4, Gemini 2.5, Qwen3 among them — across 8 input lengths on tasks held at constant difficulty. Every single model degraded as context length grew. Chroma's own framing of the finding: "Whether information is present in a context is not all that matters; what matters more is how that information is presented." Their most concrete claim: "a 1M-token window still rots at 50K tokens." A 1M-token window is not 1M tokens of equal-quality memory — it's a deeper well.

Chroma's study also surfaced a genuinely counterintuitive finding worth sitting with: logically coherent, well-written prose is worse for needle retrieval than shuffled, incoherent text. A model reading a coherent narrative "follows the story" — its attention flows with the argument, which competes with and dilutes its focus on your specific query. A shuffled haystack gives the model nothing coherent to latch onto except the one thing that actually makes sense: your needle. This is the direct justification for the engineering fix below — strip connective tissue, headers, and narrative framing from injected documents, and present them as isolated, addressable data rather than a story.

The fix: XML structure plus query injection

This lesson's worked comparison puts the fix to the test directly. Seven near-duplicate pub-description documents — differing only in capacity and dietary label, deliberately close to each other — are fed to meta-llama/Meta-Llama-3.1-8B-Instruct two ways: as a plain-text dump, and wrapped in XML tags with the query repeated at both the top and bottom of the block (the "Sovereign Injection Pattern").

The Ensign Ewart holds up to 80 people and has vegan options.
The Bow Bar holds up to 45 people and has vegetarian options.
The Jolly Judge holds up to 80 people and has vegetarian options.
The Hanging Bat holds up to 45 people and has vegan options.
The Bow Bar holds up to 80 people and has vegan options.
...
Question: What is The Bow Bar's capacity, and does it have vegan options?

The plain dump collapsed two near-identical documents into one wrong answer. The XML-tagged version got it right and cited its source. Nothing about the underlying model changed between the two runs — only how the same information was presented.

This isn't decoration — it directly answers Liu et al.'s finding. Primacy bias favors what's at the top; recency bias favors what's at the bottom; repeating the query at both positions manually "lifts" the bottom of the U-curve so the middle of your context stays actionable instead of getting lost.

Closing the loop: caching the stable part of your prompt

One more engineering consequence follows directly from everything above: if your system prompt and tool schemas are large and stable across turns, sending them fresh on every call is not just an attention-dilution risk, it's a direct cost tax. This lesson's own token-tax experiment simulates a 5-turn PyNanoClaw conversation with a roughly 2,000-token system prompt and measures per-turn cost with no caching: 2,120 → 2,216 → 2,311 → 2,395 → 2,486 tokens, totaling 11,528 tokens across five turns — the vast majority of which is the same system prompt, re-processed from scratch, five separate times. Freezing that stable prefix in a KV-cache and paying for it once instead of five times is the deck's own stated fix, cutting session cost by roughly 85% in this simulation. That's the full scope of caching this lesson owns — it's the mechanism that closes out the context-engineering arc for Hour 1. The broader question of which model to run this on, and how to reason about model economics more generally, is next Saturday's lesson.