Hello JARVIS - Building Voice Interfaces for Your LLMS
summary
Live demo builds a Jarvis‑like voice UI that streams WebRTC audio via Twilio + Deepgram + OpenAI + 11Labs, pauses playback on new utterances, sends transcripts to the LLM, generates TTS and resumes — interruption handling works by passing currently saying and an interrupt flag so responses stay on target. The system plays voice interstitials from tool status callbacks, updates a Segment profile (nut allergy, milk preference) and mostly succeeds despite demo feedback and voicemail hiccups.
AI-written summary · about
chapters (10)
- 0:00Introduction and inspiration
- 2:54The evolution of voice AI
- 6:34Core components of voice AI
- 9:31Addressing the uncanny valley
- 12:55Designing natural interaction
- 16:57Live demo: Interruption handling
- 24:31Voice interstitials and status
- 33:03Voice orchestration with Twilio
- 35:40Contextual intelligence
- 39:00Final demo and conclusion
related talks
01h 04m“Hello, AI?!” – Real-Time Voice Interactions with LLMs
21mBuilding a Voice-Enabled AI Assistant With Javascript
39mLive voices with LiveView
17mBuilding Effective Voice Agents
30mLook ma, no hands! Multimodal AI-Agents in the browser
01h 02m[Full Workshop] Building Conversational AI Agents
33mDesigning Voice Agents for Real Conversations
10mBuilding Conversational AI Agents at Scale