Failure 01: Turn-Taking Is Harder Than You Think
Prerequisite: Lesson 02's orchestration-layer framing, specifically, that voice has no explicit turn boundary and the agent has to infer one from silence, which is where every failure in this course starts. After this lesson, you can: explain why ASR endpointing fails on ordinary breath pauses regardless of transcription accuracy, and why a semantic completeness check, not a tuned silence threshold, is the actual fix.
This is one of the most common voice-agent failures in production, not the most exotic, the most ordinary. If you've ever had a phone bot cut you off mid-sentence, or grab half of what you said and act on it, you've lived this one from the caller's side.
The question ASR is actually answering
ASR (automatic speech recognition, the layer that turns the caller's audio into text) finalizes a transcript when it detects a pause in the audio, a stretch of silence past some threshold. That's the question it's built to answer, and it's the wrong one for deciding whether a caller is actually done:
| The question | Type | |
|---|---|---|
| What ASR asks | "Is the audio silent right now?" | Acoustic |
| What actually matters | "Is this a complete thought?" | Semantic |
Modern ASR answers its own question very well. The problem is that those two questions have different answers in a case that happens constantly in ordinary speech: a caller pauses to breathe.
The acoustic signal for "I'm taking a breath" and the acoustic signal for "I'm finished talking" are identical. Both are silence. There is no acoustic feature that tells you which one you're looking at, because the difference isn't in the sound. It's in what the speaker intends to do next, and intent doesn't show up in a waveform.
The failure, concretely
"My number is zero six six four—" and breathes before continuing with the rest of the digits.
It crosses the silence threshold and finalizes the transcript right there, on half a phone number.
It looks up an account using four digits that were never meant to stand alone. Wrong account, wrong caller, and from the caller's side, a bot that interrupted them for no visible reason.
Notice what didn't cause this: the ASR wasn't inaccurate. It transcribed exactly what it heard, correctly. The failure isn't a transcription-quality problem, which is why "just use a better ASR model" doesn't fix it. A perfect transcriber still has no way to distinguish a breath from a full stop, because the acoustic evidence for both is the same. You cannot out-model your way past an ambiguity that isn't in the audio to resolve in the first place.
The fix: a semantic layer above endpointing
The fix is to stop asking ASR to make the completeness decision at all, and add a layer above it that asks the actual question: given what's been transcribed so far, is this a complete thought? That's a semantic judgment, not an acoustic one. It needs to look at the content of the utterance, not just the silence after it. "My number is zero six six four" is obviously incomplete as a US phone number; a layer that knows that can hold, rather than finalize, even though the ASR engine below it already flagged a pause.
Where to build it
This is becoming a model capability in its own right in 2026. Deepgram Flux ships conversational turn detection, and LiveKit has its own turn detector, both aimed at exactly this problem.
If your compliance requirements fix which ASR vendor you're allowed to use, you can't necessarily depend on that vendor also shipping the semantic layer you need. Build the "is this complete?" layer in your own harness, sitting above whatever ASR you're required to use, rather than assuming it'll always come bundled with the transcription service. That way the fix travels with you even if the ASR vendor doesn't.
A reference demo built for this course plays a caller reading a phone number with a real breath in the middle. Watch what happens: Deepgram emits two separate finalized transcripts, and a naive agent, one that treats "finalized" as "complete," commits to the first, truncated one. Then the same audio runs through a small-LLM completeness layer, which recognizes the digits don't yet form a valid number and correctly waits for the rest before acting.
Turn-taking is about when the agent commits to what it heard. The next failure looks similar on the surface (it's also about a short piece of caller speech), but it's a different mechanism entirely: not whether an utterance is finished, but what a finished, short utterance actually means.
Reply here and it goes straight to Rod. Same as replying to one of his emails.
