Lab: The Needle in Xolo's Ledger
The previous unit predicted a specific, testable shape: recall should drop in the middle of a long context, worse for a smaller model than a larger one, worse still as the reasoning required gets more complex than plain lookup. This unit stops predicting and runs the test, against Xolo's own data, with one of the seven seeded errors playing the needle. By the end you'll have watched the U-shaped curve happen on your own machine, not read about it happening somewhere else.
This is the lab unit of Class 4. It's meant to be run, not just read: build the synthetic transaction log, plant one seeded error at a chosen position, and watch whether the model finds it. The next and final unit turns what you learn here into a structural defense.
The setup: one real Xolo error, buried on purpose
The classic version of this demo, in the source material this course draws from, buries a made-up secret sentence inside a wall of unrelated prose and asks a model to retrieve it. That version proves the mechanism, but it proves it about nothing Xolo's organization will ever actually do. The version that matters to this course buries something real: one of the seven seeded errors from Class 3, specifically the currency-mismatch deposit, the one that came in as US dollars instead of pesos, inside a long, synthetic transaction log shaped like Xolo's actual books, and asks the same question a reconciliation agent has to answer honestly every month: does anything in this file not add up.
Of Xolo's seven traps, the currency mismatch is the cleanest needle for this experiment. It's a single, unambiguous row, one deposit tagged USD where every other row is MXN, so whether a model caught it is a binary, checkable fact, not a matter of degree. The duplicate invoice or the client-name-spelling trap would work too, but they involve comparing two rows against each other rather than recognizing one row as anomalous on its own, which adds a second variable this experiment doesn't need yet.
Building the haystack
The transaction log is synthetic, generated on purpose, the same design choice Class 3 made for Xolo's core data, and for the same reason: you need to know exactly where the needle is to measure whether the model found it.
# build_haystack.py — a synthetic transaction log shaped like Xolo's real data,
# with one seeded error (the USD-instead-of-MXN deposit) planted at a chosen position
import csv
import random
random.seed(2026)
CLIENTS = ["Aguilar y Asociados", "Ferretería del Norte", "Studio Reyna",
"Grupo Peninsular", "Café Cardamomo", "Talleres Ibarra"]
def make_row(row_id, seeded=False):
amount = round(random.uniform(3000, 45000), 2)
currency = "USD" if seeded else "MXN" # the needle, when seeded=True
return {
"row": row_id,
"client": random.choice(CLIENTS),
"amount": amount,
"currency": currency,
"date": f"2026-07-{random.randint(1, 28):02d}",
}
def build_haystack(total_rows, needle_position, path="haystack.csv"):
rows = []
for i in range(1, total_rows + 1):
seeded = (i == needle_position)
rows.append(make_row(i, seeded=seeded))
with open(path, "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=rows[0].keys())
writer.writeheader()
writer.writerows(rows)
return rows
# three haystacks, same needle, three positions: start, middle, end
build_haystack(200, needle_position=5, path="haystack_early.csv")
build_haystack(200, needle_position=100, path="haystack_middle.csv")
build_haystack(200, needle_position=196, path="haystack_late.csv")Two hundred rows is deliberately modest, small enough to fit comfortably in any model's context window with room to spare, which matters: this experiment is not testing whether the file overflows the window, Class 4's first unit already covered token limits as a separate concern. It's testing whether a fact that fits easily still gets missed, purely because of where it sits. That's the actual claim "lost in the middle" makes, and it's worth isolating cleanly rather than confusing with a simpler, less interesting failure.
Running the test, three positions, two models
The brief stays identical across every run, only the file and the model change, exactly the discipline of changing one variable at a time that makes the result mean something.
Reconcile haystack_{position}.csv. Flag any row whose currency
is not MXN. Report the row number and amount of anything flagged.
Point a small, fast model tier at each of the three files with the identical brief above. This is a deliberate choice, not a convenience: a smaller model has less capacity to fight attention dilution, per the previous unit's prediction, so it's the model most likely to make the U-shaped curve visible rather than washing it out with brute capability.
Repeat the identical three runs against the largest tier available. The prediction from the previous unit says a frontier model should show the same U-shape in principle, softmax divides a fixed budget regardless of model size, but recover from it more gracefully, catching the middle-positioned needle more often than the small model does, even if not as reliably as the edge positions.
Six cells: two models times three positions. Each is a simple binary, did the flagged-row report correctly name the seeded row, yes or no. Resist the urge to average or round this away. The whole point of the experiment is visible in the raw grid, not in a summary statistic.
What a typical run actually shows
Expect something close to this shape, though exact behavior varies by model version and this course's standing advice from Class 3 still applies: trust what you actually observe on your own run over any specific numbers printed here. The small model reliably catches the needle at the early and late positions and measurably degrades in the middle, sometimes missing the flagged row entirely, sometimes reporting it with lower confidence phrasing even when it does surface it. The frontier model catches the early and late positions just as reliably and, critically, still shows some falloff in the middle, a real but smaller dip rather than the small model's larger one.
Naming the choice: why a small model belongs in this lab on purpose
It would be easy to run this whole experiment only against a frontier model and call it done, especially since a frontier model is what the rest of this course, and Xolo's real organization, actually runs on day to day. That would be the wrong call for this specific lab, and it's worth being explicit about why, per this course's standing rule against silently reaching for a weaker tool without saying so out loud: a small, comparatively weak model is used here deliberately, specifically because its failure mode is the one that makes the underlying mechanism visible without ambiguity. A frontier model's greater capacity can partially mask attention dilution, which is scientifically real progress but pedagogically inconvenient for a first look at the effect. Seeing a weaker model fail cleanly, then watching a stronger model fail the same way but less, is a cleaner lesson than only ever watching a model that mostly succeeds.
Unit 1's plan-with-a-big-model, execute-with-a-small-one strategy was about matching capability to a step's actual difficulty in production. This lab's use of a small model is different: it's a deliberate research choice to expose a mechanism clearly, not a claim that Xolo's real reconciliation skill should run on a weak model. Don't confuse a teaching instrument with a production recommendation; the skill from Class 3 still calls the tier appropriate to the step, exactly as Unit 1 argued.
What this means for how Xolo's skill actually reads its data
The reconciliation skill from Class 3 already avoided this failure by construction, not by luck, and this lab is the first point in the course where you can see exactly why. It never pastes a raw, unordered transaction log into a single open-ended prompt and hopes the model notices anomalies. It works from an explicit brief with named control totals, forcing verification of specific facts, every deposit's currency, every invoice's collection status, rather than open-ended review of an undifferentiated pile. A control-total check for "sum of MXN deposits must equal total collected revenue" doesn't care where in the file the offending USD row sits, because the check isn't relying on the model noticing it during a read-through. It's relying on arithmetic that fails loudly regardless of position. That's the real payoff of Class 3's design, made visible now with a mechanism to explain why it mattered.
Where this leaves you
You've now run the lost-in-the-middle effect for real, against a synthetic version of Xolo's own transaction data, with a real seeded error standing in as the needle, and watched a real U-shaped result, or something close to it, come out of your own terminal rather than out of someone else's paper. You've seen a small model's failure make the mechanism legible, and a frontier model's smaller but nonzero failure confirm it isn't a capability gap alone. And you've connected the result directly back to why Class 3's control-total design already defended against this failure, before this course ever explained the mechanism behind it.
The final unit of this class turns that observation into a general defense: brackets and XML as a way to structure a document so a model's attention lands where it should regardless of position, why that same structuring doubles as a defense against prompt injection, and how rotating a document set pays off the near-duplicate-client-name trap this course planted back in Class 1.
Structuring Xolo's own contracts so nothing gets lost in the middle, and so an injected instruction has nowhere to hide.
Reply here and it goes straight to Rod. Same as replying to one of his emails.
