10 Things I Learned Building Voice Agents in 2025
summary
Builds voice agents using Agora OpenAI 11 Labs Cerebrus Pinecone, finds cascading pipelines beat all‑in‑one for latency with a 650 milliseconds cascading flow but need custom routing because each server handles only 3–4 users per server when an agent stays 'live'. Encounters mismatched audio vs transcript, web sockets lag vs UDP, streaming incompatible with function calling, RAG accuracy limits and maintenance, and solves PSTN ordering issues with a PSTN multi-agent ordering approach that works but requires careful prompts and audit logging.
AI-written summary · about
chapters (14)
- 0:00Building voice agents intro
- 1:24Voice agents scaling challenges
- 3:10Load balancing for voice agents
- 4:53Agora conversational AI engine
- 6:40UDP vs web sockets for voice AI
- 7:45Child-safe voice agent guardrails
- 9:11Prompt engineering prevents jailbreaks
- 10:57Customizing agents without backend
- 11:39Long prompts and hallucinations
- 12:43Function calling needs full responses
- 14:47Clear function call examples
- 15:30Ordering dinner with PSTN
- 18:44Resources for voice agents
- 19:41Demo troubleshooting & closing
related talks
33mDesigning Voice Agents for Real Conversations
56mAI Agents That Don’t Suck: How to Build Ones That Actually Drive Business Value
01h 26mBuilding voice agents with OpenAI
17mBuilding Effective Voice Agents
01h 02m[Full Workshop] Building Conversational AI Agents
36mHow We Built a Voice Agent with Temporal
52mDesigning Production-Ready Agentic Systems: Lessons from the Field
14mEmpowering Voice AI Agents with Real-Time Knowledge from Neo4j