Engineering voice agents: Latency, quality, and scale
summary
Concrete numbers and architectures decide whether to watch: speech-to-text hits P90 <100ms, ~6% WER, LLMs must reach TTFT 200–300ms, 8–30B models so tool calling stays reliable, and text-to-speech needs low time-to-first-audio and real-time-factor <1. Network hops add 75ms and wipe out budgets unless systems co-locate to drop 75ms→5ms, and the practical pattern that proves useful is a tiny “thinker” LLM that routes one heavy tool call to a larger model to keep the fast path responsive.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:19Voice agents today
- 2:41Voice agent challenges
- 4:28Pipeline architecture
- 5:10Speech-to-text metrics
- 8:40LLMs as brains
- 9:44Text-to-speech quality
- 11:09Latency and cost trade-offs
- 12:11Auto-scaling deployments
- 13:13Network latency strategies
- 14:37Speech-to-speech future
- 16:42Q&A on evals
- 18:27Q&A on co-location
- 19:30Q&A on guardrails
- 21:38Q&A on voice evals
- 22:41Closing remarks
related talks
33mDesigning Voice Agents for Real Conversations
17mBuilding Effective Voice Agents
13mVoice In, Visuals Out: The Agony and the Ecstasy
36mHow We Built a Voice Agent with Temporal
13mHow We Reduced Latency by 70% While Maintaining 99% Accuracy
20m10 Things I Learned Building Voice Agents in 2025
56mAI Agents That Don’t Suck: How to Build Ones That Actually Drive Business Value
17mServing Voice AI at Scale