Building an AI assistant that makes phone calls
summary
Workshop shows a working voice assistant that makes live phone calls, streams audio to Google Cloud for realtime transcription, routes conversation through GPT-4 and sends synthesized replies back via Twilio, then writes actions and transcripts into a Convex-backed database and triggers calendar or SMS updates. Codebase is Typescript with Node Express and Next and a public repo. Prototype works end to end as a proof of concept but notes measurable latency and optimization needs. Convex vector database Google Cloud speech-to-text Twilio and GPT-4 API
AI-written summary · about
chapters (7)
- 0:00Building smarter AI assistant
- 3:56Floyd’s high-level architecture
- 11:48Live demo: Floyd calls
- 20:46Convex simplifies backend
- 31:17Debugging and scaling
- 42:34Latency in real-time calls
- 46:32Q&A
related talks
21mBuilding a Voice-Enabled AI Assistant With Javascript
56mAutomating Voice Assistants and Chatbots
01h 04mHow I used AI voice to help me play Magic: The Gathering
01h 26mBuilding voice agents with OpenAI
44mDeveloping Skills & Actions for Voice Assistants
01h 05mHow I used AI voice to help me play Magic: The Gathering
40mBuild advanced Voice Assistants and ChatBots
44mSmartly AI: Cross Platform Voice Application Creation