Voice Agents, Not Chatbots
A voice agent that reads perfectly in a chat transcript can still fail on a real call, because voice breaks in places text never tests: who's allowed to interrupt whom, whether a pause means finished or thinking, what happens when three concurrent systems disagree about what just got said. This course names the six specific ways that happens, starting from two measured numbers about how fast humans actually take turns in conversation, and hands you five reusable orchestration patterns that fix all six without waiting for a smarter model. You'll build a two-tier classifier that tells an interruption from a backchannel, design cooperative cancellation so 'stop' actually stops every in-flight operation, and leave knowing the two regulatory regimes and the voice-specific metrics that predict abandonment before a customer ever complains. Migrated from a Rasa/TMLS 2026 conference talk, with a worked reference demo behind every failure.
MethodThis course teaches a discipline. The examples use today's tools; the method is meant to outlast them.
Rod Rivera
Professor
Desk
First lesson is free to preview.
What You'll Learn
- Explain, using two measured timing-research numbers, why voice agents fail on architecture, not model speed
- Distinguish cascaded (STT-LLM-TTS) from speech-to-speech voice architectures and state why regulated industries stay cascaded in 2026
- Fix acoustic-vs-semantic turn-taking failures with a completion layer that sits above raw ASR endpointing
- Build a two-tier classifier that correctly reads the same word as an interrupt or a backchannel depending on context
- Design cooperative cancellation so a user's "stop" actually stops every in-flight operation, not just the one closest to the microphone
- Apply progressive control — agentic by default, deterministic hard gates at the seams that matter — to call-ending, handoff, and compliance disclosure alike
- Make state, not the model, the only source of truth a voice agent is allowed to speak aloud
- Design a typed handoff envelope so a transfer carries context, not just audio
- Name the two regulatory regimes governing voice-agent disclosure and the voice-specific metrics (p95, false barge-in rate) that predict abandonment
- Trace all six named failures back to the five reusable orchestration patterns that fix them, none of which wait on a smarter model
Prerequisites
- Comfort with the idea of an LLM calling tools; this course does not re-teach tool-calling mechanics from scratch
- Helpful, not required: `Tools, Memory, and Multi-Agent Systems` — lesson 02 of this course cross-links to that course's own cascaded-vs-speech-to-speech treatment rather than re-deriving it
- No prior voice-agent or telephony experience required — ASR, TTS, and SIP are each introduced from their role in the failure they cause, not from first principles
Syllabus
Ready to keep going?
The lessons in Voice Agents, Not Chatbots are open. The Prof Rod weekly lesson is the next thing, not a door.
