Failure 05: Tool Calls the Agent Has Already Spoken
Prerequisite: Lesson 06 ("Failure 04: The Agent Says Goodbye Too Soon") and progressive control — agentic by default, deterministic hard gates at the seams that matter. After this lesson, you can: explain why a hallucinated number is a worse failure in voice than in chat, and build the state-as-source-of-truth pattern that makes the LLM structurally incapable of speaking a figure it invented.
Lesson 06 put a deterministic gate at the seam where the agent decides to end a call. This lesson is about a different seam — the moment the agent decides what number to say — and why, in voice, that seam has to be gated even more strictly than in chat.
The same hallucination, a different medium
Hallucinated tool-call arguments are a known problem in chat: the model invents a figure, calls a tool with it, and the wrong number appears in a text response. Voice looks like the same failure, but the two aren't equally dangerous:
| Chat | Voice | |
|---|---|---|
| Where the wrong number lands | A chat window, as text | Spoken aloud, on a recording |
| Catch window | Reviewable before the user acts on it or a human catches it | None, TTS has already rendered it by the time anyone could check |
| Undo | "Let me re-check that" | Doesn't un-hear anything |
Same underlying model failure, different medium. Chat gives you a chance to catch the error before it lands on the customer; voice doesn't.
This isn't a hypothetical harm. For the full facts of Moffatt v. Air Canada (BC Civil Resolution Tribunal, February 2024), the invented refund policy, Air Canada's "separate entity" defense, and the tribunal's ruling, see why-agents-fail lesson 02, which carries the fullest citation of the case on this site, including the tribunal's own words. What that case establishes is liability for what a bot says. What this lesson adds is why voice makes that liability land faster and harder than chat does: there's no gap between the agent speaking and the customer hearing where anyone, human or system, could have caught the number first.
The fix: invert who is allowed to speak the truth
The naive path is direct and dangerous: the LLM emits a number, a tool runs on that number, TTS speaks it. One bad token and the customer has heard a refund amount that doesn't match your ledger.
The fix inverts the responsibility. The LLM is still allowed to decide which tool to call; that's a reasoning task it's good at. It is never allowed to decide what the arguments are.
Tool selection stays with the model, that's the reasoning task it's actually good at.
The tool's arguments come from validated session state the application controls, never from anything the model generated inline.
Neither reads from the model's raw output. The model becomes structurally incapable of speaking a number it computed itself, because the number it speaks was never its own.
State-as-truth stops a hallucinated number from reaching the customer's ears. It doesn't by itself stop a correct number from triggering an irreversible action too casually. For refunds, transfers, or anything that can't be undone, add a spoken confirmation gate on top: the customer has to explicitly trigger the irreversible step, rather than having it fire on a generated response alone.
The line to hold onto: the LLM picks the tool, never the truth.
Hearing the difference
A reference demo built for this course runs a naive agent eight times with every spoken figure checked against the ledger, and occasionally you'll hear it happen — a wrong refund amount, synthesized and spoken aloud, no different from a correct one to the customer's ear. Then it runs the state-as-truth version: the wrong number isn't caught after the fact, it's structurally impossible to reach the TTS layer at all, because the TTS layer was never given anything the model wrote itself.
What to build
Draw a hard line between what the model is allowed to decide and what it's allowed to say. Tool selection stays with the LLM. Tool arguments come only from validated session state the application controls. TTS reads from that same state, never from the model's raw output. Add a spoken confirmation gate wherever the action is irreversible. None of this asks the model to be more careful; it makes carelessness structurally unable to reach the customer.
State-as-truth keeps a hallucinated number from ever reaching the customer's ears. The next failure happens after the call goes right — when a human takes over and the agent hands off nothing.
Reply here and it goes straight to Rod. Same as replying to one of his emails.
