Tokens, Generative Models, and What a Model Actually Costs
Three classes in, you've briefed Claude Code, watched it harden into an organization, and closed Xolo's scoreboard at seven of seven. All of that ran on faith in one sentence: a model predicts the next token. This class opens that sentence up. Not to make you a machine-learning researcher, you don't need the math behind the weights, but to make "the model" stop being a black box you trust by reputation and start being a mechanism you can reason about, the same way last class turned "the tool" into four terms of an equation you could debug.
This is the first unit of Class 4. It covers what a generative model actually does mechanically, how text becomes the tokens a model bills and reasons in, and the real economics of choosing which model tier does which part of Xolo's work. The next three units build directly on this one: forgetting in a long context, a live needle-in-a-haystack lab, and structuring documents to survive both.
Generative, not discriminative: the distinction that explains the word "hallucination"
Most of the machine learning you may have encountered before this course, a spam filter, a fraud flag, a credit score, is discriminative: it looks at an input and sorts it into a category or a number. Is this email spam, yes or no. Is this transaction fraudulent, probability 0.03. A discriminative model draws a boundary through existing categories and tells you which side something falls on.
Claude is a different kind of thing: a generative model. Instead of sorting an input into a pre-existing bucket, it produces new output, one token at a time, by estimating the probability distribution over "what plausibly comes next," given everything that came before it in the context window. Nothing about that mechanism has a built-in concept of "true." It has a mechanism for "probable, given this exact context."
That single distinction is the honest answer to why a model can state something confidently wrong. A discriminative fraud model that's uncertain outputs a low-confidence score, an admission of uncertainty baked into its very output format. A generative model asked "what invoice number does this deposit correspond to" doesn't have a native "I don't know" bucket the way a classifier does. It has a next-token distribution, and if nothing in its context sharply favors "I don't know" over a plausible-sounding invoice number, it will generate the plausible-sounding one. This isn't a bug report on any particular model. It's the shape of the mechanism, and it's the actual reason Class 3 spent an entire class on evidence requirements and independent verifiers instead of trusting a model's own confident tone.
Next-token prediction, made concrete
Here's the mechanism in full, because "predicts the next token" is a phrase people repeat without ever watching it happen. Given a sequence of tokens, the model produces a probability distribution over every possible next token in its vocabulary, tens of thousands of candidates, each with a probability. One gets selected, according to a sampling strategy, appended to the sequence, and the whole process repeats, one token at a time, until a stop condition fires.
Picture it running against Xolo's own reconciliation report. The model has already generated The gap between crm_invoices.csv and bank_deposits.csv totals and now needs the next token. The candidates and probabilities below are illustrative, invented for this walkthrough rather than pulled from a logged model call, but the shape is real: candidates might include 12,847 (probability 0.41, because the actual computed total is close to this), $ (probability 0.19, if a currency symbol plausibly comes first), zero (probability 0.02, an unlikely but not impossible continuation), and thousands of far less probable tokens trailing off toward zero. The model samples from that distribution, appends the winner, and asks the same question again for the token after it. There is no step in this loop that consults Xolo's actual bank statement to check whether 12,847 is correct. The number's plausibility, not its correctness, is what won the token. That's exactly why the reconciliation skill from Class 3 was built to recompute control totals independently rather than trust a number generated this way.
Tokenization: fragments, not words, and why it isn't free
A token is not a word. It's a fragment, somewhere between a character and a word, chosen by a tokenizer trained to compress text efficiently. reconciliation might split into re, concil, iation, three tokens for one word. MXN might be one token. A number like 48,392.17 frequently splits into several tokens, digit groups and punctuation each costing their own slot. This matters for two concrete reasons, not as trivia.
First, cost. Every API call to a model is billed by token, input and output separately, and the two are priced differently. Output tokens typically cost more per token than input tokens across current frontier providers' published pricing pages; check the specific provider's own rate card before budgeting a real run, since exact ratios shift as pricing changes. A prompt that pastes Xolo's entire data/crm_invoices.csv inline costs real money in input tokens whether or not the model needs all of it for the current question, and a verbose, hedging answer costs more in output tokens than a terse one that says the same thing. Second, and less obvious: tokenization varies by language and by content type. Code, dense numeric data, and non-English text often tokenize less efficiently than plain English prose, meaning the same underlying information can cost meaningfully more or less depending on its shape. A CSV full of peso amounts and client names is not a token-cheap format compared to the same information summarized in prose, which is worth knowing before assuming "just paste the whole file in" is a free move.
/cost, the command from the reference manual two classes ago, reports exactly this: tokens spent, translated into dollars. Nothing about tokenization was hidden from you then. This unit just opens the hood on what that number was actually counting all along.
Estimating that cost ahead of a run, rather than only reading it afterward from /cost, is a two-line habit worth having before you brief anything expensive against Xolo's files.
# rough token-cost estimate before running a reconciliation brief
import csv
def estimate_input_tokens(path, chars_per_token=4):
with open(path) as f:
raw = f.read()
# crude but usable: real tokenizers vary by ~15-20% from this
return len(raw) // chars_per_token
crm_tokens = estimate_input_tokens("data/crm_invoices.csv")
bank_tokens = estimate_input_tokens("data/bank_deposits.csv")
print(f"crm_invoices.csv: ~{crm_tokens} tokens")
print(f"bank_deposits.csv: ~{bank_tokens} tokens")
print(f"pasting both in full: ~{crm_tokens + bank_tokens} input tokens, every call, uncached")The four-characters-per-token rule of thumb is deliberately crude, real tokenizers vary by content, but it's precise enough to catch the actual mistake this section warns about: pasting two CSVs in full, on every single call, when a cached, well-scoped brief would touch a fraction of that.
Model-selection economics: Xolo's reconciliation as the worked case
Every frontier model family ships in tiers, roughly: a small, fast, cheap model; a mid-sized balanced one; and a large, slow, expensive one with the deepest reasoning. /model from Class 3's reference manual is the lever that picks among them for a given session. The question this section actually answers is when each tier is the right call, using Xolo's own reconciliation task as the worked case rather than an abstract cost table, because you already know this task's exact shape: match data/crm_invoices.csv against data/bank_deposits.csv, catch all seven seeded errors, produce a report with control totals.
Break the task into its actual steps and the tiering answer falls out of the work itself, not out of a rule of thumb.
Deciding how to structure the reconciliation, which join key to use, which control totals prove completeness, how to handle the currency mismatch once it's found, is exactly the kind of multi-step reasoning where a larger model's deeper capability shows up as real accuracy, not just a nicer writing style. Getting the plan wrong here is expensive precisely because everything downstream inherits the mistake, the same "ten seconds versus twenty minutes" arithmetic Class 3 built around plan mode.
Once a plan exists, actually walking data/crm_invoices.csv row by row and matching against data/bank_deposits.csv on invoice ID is comparatively mechanical: apply the rule, flag the exception, move to the next row. A smaller, faster, cheaper model handling well-specified, low-ambiguity execution steps produces output just as reliable as a frontier model here, at a fraction of the token cost, because the hard reasoning already happened in the planning step.
The verifier subagent from Class 3, checking the report against raw data independently, is asking a judgment question again, does this control total actually explain the gap, is this exception real or a false positive, so it benefits from the same tier that did the planning. It's a short, bounded call, not the whole reconciliation rerun, so the cost stays small even at the higher tier.
| Reconciliation step | Right tier | Why |
|---|---|---|
| Planning the join key, exception rules, currency handling | Large | Ambiguity-resolving reasoning; mistakes here propagate downstream |
| Matching rows against the plan | Small | Mechanical execution against an already-clear rule |
| Verifying control totals independently | Large, brief call | Judgment again, but bounded in scope |
That three-step shape has a name: plan with the expensive model, execute with the cheap one, and it's worth being explicit about why it isn't just "use the cheap model whenever you can get away with it." The failure mode of routing everything to the cheap tier is invisible until it isn't: a naive small-model run might handle the mechanical matching fine and then silently mishandle the client-name-spelling trap or the currency mismatch, the exact two traps Class 3 showed getting missed by an under-specified brief regardless of which model ran it. Tiering saves money on the steps where capability doesn't change the answer, and spends it deliberately on the steps where it does.
A cheap model given a vague brief fails the exact same way a frontier model given a vague brief fails, missing traps silently, per Class 3's naive-run autopsy. Model tier doesn't substitute for a five-part brief. It's a separate lever, and the two compound: a well-briefed cheap model executing a well-planned step is reliable and inexpensive; a badly briefed model at any tier is neither.
Prompt caching: paying once for what repeats
Xolo's CLAUDE.md, its skills, and its raw data/ files don't change between one reconciliation run and the next run five minutes later, but a naive integration re-sends all of it, in full, as input tokens, on every single call. Prompt caching is the fix: a provider-side mechanism that recognizes a repeated prefix, the unchanging system prompt, the unchanging CLAUDE.md, the unchanging tool definitions, across calls, and charges a small fraction of the normal input-token rate for the cached portion instead of the full rate, while only the actually new part of the request, this turn's real question, gets billed in full.
For a task shaped like Xolo's monthly reconciliation, run repeatedly against a large, mostly-static CLAUDE.md and skill set, caching isn't a minor optimization. This is the general shape of the effect, not a measured figure for Xolo's own workflow specifically, but for a task this repetitive it can plausibly be the majority of the cost difference between a workflow that's cheap enough to run on every file change and one that isn't. This is also a second, independent reason CLAUDE.md stays lean, per Class 3's warning against the four-hundred-line handbook: every line in it is either riding the cache at a steep discount on repeat calls, or, on the very first call of a session, costing full price once. A bloated handbook is expensive twice over, once in relevance and once in the token bill for the first call that has to pay full freight before the cache exists.
Where this leaves you
You now have a mechanical account of what "the model" actually is: a generative process estimating a probability distribution over the next token, with no native concept of truth built into that mechanism, which is the real reason this course has spent three classes on evidence and independent verification rather than trust. You've seen tokens as the actual unit of cost, not a technicality, and watched Xolo's own reconciliation task carry the argument for tiering models by step rather than treating "model choice" as one global decision. And you've seen why prompt caching turns a stable CLAUDE.md from a line-item cost into a discount, tying directly back to the lean-handbook discipline from two classes ago.
None of this yet explains what happens when a task's context grows long, when Xolo's transaction log runs to hundreds of rows and the model has to hold all of it in view at once. That's the next unit: what a context window actually does to the tokens sitting inside it, and why the middle of a long document is measurably the least reliable place to bury anything that matters.
Reply here and it goes straight to Rod. Same as replying to one of his emails.
