Prompt Caching and Model Economics

Ask an agent to "find a pub in Edinburgh for 160 people with vegan options" and one model has to decide how to break that goal down, while a different call has to confirm the booking actually happened. Same task, two very different jobs — and this lesson's labs show that running both on the same model, at the same size, is where the money and the correctness both leak out.

Lesson 02 closed with a caching fix: freeze your stable system prompt in a KV-cache and stop paying to re-process it every turn. That's the mechanism. This lesson picks up the question caching doesn't answer — once your context is engineered well, which model do you actually run it on, and does the answer change turn to turn within the same agent?

Most engineers treat model selection as a single decision made once, at setup: pick the best available model, wire it in, move on. This lesson's own labs argue that's the wrong frame. The better question is per-role, per-turn: what does this specific step need — deliberation, or speed? — and the answer routinely differs within a single agent loop.

The Planner-Executor split, and where it breaks

The architecture this lesson builds toward: a "thinking" model deliberates on the goal and writes a precise, multi-step plan (the Planner); a fast, cheap model carries out each individual step (the Executor). The lab wires this up directly against the Edinburgh problem — Qwen/Qwen3-235B-A22B-Thinking-2507 as Planner, meta-llama/Meta-Llama-3.1-8B-Instruct as Executor.

python planner_executor_lab.py

The Planner held up exactly as designed: given "Find a pub in Edinburgh for 160 people with vegan options," it decomposed the goal into three coherent, sequenced steps without dropping any of the three constraints. That's what the expensive model earns its cost doing. The Executor is where the lab gets interesting — and where the misconception this lesson corrects becomes visible. Step 3 asked the Executor to confirm a booking, and the model's response claimed to have "secured a reservation." No tool call was made. No pub was contacted. The 8B model followed its instruction faithfully and produced confident, well-formed prose — but without a real tool wired to it, "execution" was narration, not action.

That gap is deliberate scope for this lesson, not an oversight: tool wiring — how a model's text output actually becomes an executed action — is next Saturday's material in full. What this lesson owns is upstream of that: choosing the right model and reasoning strategy for a given role and turn.

Bigger is not the safer default

The instinct to "upgrade" a struggling agent by swapping in a larger model gets tested directly by this lesson's constraint-saturation lab — nicknamed the "Triple-Trap." Three models — Meta-Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, and Qwen3-235B-A22B-Instruct-2507 — are given an unsatisfiable search: a pub in Edinburgh with capacity ≥ 160, vegan options, and a quiet room, when no candidate pub in the list actually has all three. The task is designed to force a trade-off, and the honest move is to pick the closest match and say plainly which constraint failed.

All three models were honest — every one correctly flagged all_pass=false rather than lying about a match. Honesty was not the differentiator. The 8B model picked The Bow Bar, sacrificing capacity — the hardest, most quantitative constraint — to satisfy two boolean flags instead. The 70B and 235B both identified The Albanach as the better trade-off, correctly treating capacity as the constraint that's hardest to substitute around. And notably, the 235B was also the fastest of the three at 0.33s — a reminder that parameter count and latency don't move in lockstep across model families, so speed alone tells you nothing about which model to trust with a reasoning task.

The lesson here isn't "bigger models reason better," stated generically — it's narrower and more useful: correctness under saturation is what should drive model choice, not size as a proxy for it, and a bigger model's output still needs verifying, not just trusting. The 8B didn't lie. It sent 160 people to a pub that fits 80 and called it the best available option. In production, that books the wrong venue.

Self-consistency is not a fix for a systematic bias

A second common instinct: if a model's answer might be wrong, sample it multiple times and take a majority vote. This lesson's Self-Consistency lab (N=7, temperature 0.8) tests that instinct on an inverse-proportion word problem — correct answer: 4 — using Meta-Llama-3.1-8B-Instruct.

Five of seven samples confidently returned the wrong answer, and majority vote amplified that error instead of correcting it. The distinction this result draws: self-consistency helps against random token drift — the chance a model derails on a low-probability token mid-reasoning. It does nothing against a systematic bias — here, the model consistently confusing inverse and direct proportion. Voting seven wrong-leaning samples doesn't produce a right answer; it produces a confident wrong one. If a model gets a problem wrong most of the time it tries, more votes won't fix that — a better reasoning strategy or a stronger model will.

When elaborate prompting stops paying for itself

The ACL 2025 Outstanding Paper (Liu et al., cited by name in the source deck) reports a run across 1,440 configurations, 6 models, and 8 prompting strategies — the model list and exact configuration count come from the source deck and have not been independently re-verified against the published paper, so take the 1,440 figure as the deck's own reporting rather than a number this lesson re-derived. What the run found is a reversal worth knowing before you reach for the fanciest-sounding technique: at N=1 (a single sample), Tree of Thoughts outperformed plain Chain-of-Thought — 78% versus 72%. But scaled to N=100 with majority voting, CoT reached 88% (a 16-point gain) while ToT reached only 83% (a 5-point gain) — CoT overtook it. The elaborate scaffolding that helps at low compute becomes a liability at high compute, because its extra branching instructions are just more tokens the model can misinterpret, while the diversity ToT tries to force explicitly, simple sampling already achieves implicitly once N is large enough.

The efficiency-versus-scale contrast this lesson draws alongside it: greedy decoding (N=1) on GPT-4 (175B) scored 87%, while self-consistency (N=100, majority vote) on Llama-3 (7B) scored 92% — a model 25 times smaller outperforming the larger one purely through deliberation rather than raw parameter count. This is Snell et al.'s (DeepMind, 2025) inference-time compute scaling formalized as a design choice: compute spent at inference time is fungible with compute spent at training time, and inference compute is often 10–100x cheaper than moving up a model tier.

Prompt: "Think step by step and explain your reasoning fully."

Output (~156 tokens, 8.2s):
"To find the average speed for the entire journey, we use the
formula: Average Speed = Total Distance ÷ Total Time. The train
travels 240 miles then 180 miles, giving a total of 420 miles..."

Accuracy on GSM8K: 87.2%

Chain-of-Draft (arXiv:2502.05381) reaches the same correct answer — 84 mph, in this lesson's own worked train-speed comparison against Qwen/Qwen3-235B-A22B-Instruct-2507 — at roughly a 13x reduction in tokens on the paper's benchmark tasks, and higher measured accuracy on GSM8K, not lower. The decision rule this course teaches: Chain-of-Draft for reasoning steps only your code will read (routing, tool-call construction), standard Chain-of-Thought when a human needs to follow or audit the logic.

Forcing structure instead of hoping for it

One last piece of model economics worth closing on: when your agent's output feeds directly into code, "mostly valid JSON" is a production failure waiting to happen. This lesson's strict-schema lab defines a JSON schema with strict: true and additionalProperties: false, and runs it against Qwen/Qwen3-235B-A22B-Instruct-2507:

{
  'name':     'The Thistle & Sprout',
  'address':  '123 Grassmarket, Edinburgh, EH1 2HD',
  'capacity': 160,        # integer, not "160"
  'vegan':    True        # boolean, not "true"
}

The mechanism behind that guarantee is a logit filter, not post-hoc validation: at each sampling step, any token that would violate the schema — a stray quotation mark where an integer is expected, an unrequested extra key — has its probability masked to zero before the model can choose it. The output isn't "usually" correct. It's mathematically unable to be otherwise.

What to do with this

Model selection isn't a single decision you make once and forget. It's a per-turn, per-role budget, and this lesson's labs converge on the same three-step routine each time you add a new role to an agent loop: