Building Effective Voice Agents
summary
Real-time speech-to-speech API that combines transcription, LLM reasoning, and TTS reduces end-to-end latency and preserves conversational context compared with chained STT→LLM→TTS setups, enabling low-latency production voice apps. Patterns include delegating complex tasks to smaller models via the agents SDK, summarizing state between handoffs, and limiting toolset size. Eval practices: observability, human-labeled transcripts, transcription-based rubrics, synthetic conversations, and using GT4-audio to check tone. Deploy guardrails async with configurable debounce (example: 100-character checks). Lessons from Lemonade and Tinder emphasize early evals and brand customization.
AI-written summary · about
chapters (9)
- 0:00Introduction and overview
- 1:48Audio models at tipping point
- 4:58Speech-to-speech architecture
- 7:23Key trade-offs in building
- 8:25Voice agent patterns
- 10:09Prompting and voice customization
- 11:56Tool use and delegation
- 12:59Evaluation and guardrails
- 15:52Learnings and wrap-up
related talks
33mDesigning Voice Agents for Real Conversations
01h 04mBuilding Expertise Into Generated Conversations
25mEngineering voice agents: Latency, quality, and scale
36mHow We Built a Voice Agent with Temporal
01h 04mBuilding Expertise Into Generated Conversations
56mAutomating Voice Assistants and Chatbots
56mAI Agents That Don’t Suck: How to Build Ones That Actually Drive Business Value
10mBuilding Conversational AI Agents at Scale