Orchestrating Real-Time Multimodal AI Agents with Rust
summary
A Rust-based orchestrator (Echo server / Echo Kit) glues VAD, ASR (Whisper-compatible), LLMs, and TTS into a streaming pipeline that runs on an affordable ESP32 edge device and offloads heavy inference to servers or cloud APIs like Google Gemini Live or local models such as LlamaEdge. The deployed Docker image is ~100 MB, Grock returns first token in ~100 ms and 1,000–2,000 tokens/s, VAD drops to ~0.5 s, and end-to-end latency is optimized from ~6–9 s toward under 2 s with streaming, KV caching, WebSocket/WebRTC and Agora.
AI-written summary · about
chapters (18)
- 0:00Rust and WebAssembly AI
- 1:05Customizable multimodal AI agent
- 3:42Custom voice AI agent pipeline
- 5:28Live demo: voice cloning
- 7:20Local AI agent deployment modes
- 9:11Hybrid real-time voice AI
- 11:27Rust orchestrator server
- 13:22YAML model configuration
- 15:14Multimodal pipeline explained
- 17:45Swapping AI service providers
- 18:57MCP demo: exam prep
- 21:11Tool calls with MCP servers
- 21:55Latency optimization in voice AI
- 24:08Faster model services
- 26:17Human reaction time latency
- 27:01Community projects with Echo Kit
- 28:55Future plans with museums
- 29:37Closing and resources
related talks
22mVoice Agents That Think While They Talk
30mLook ma, no hands! Multimodal AI-Agents in the browser
01h 07mBuilding and Scaling an AI Agent Swarm of low latency real time voice bots
27mWhat happens when AI starts grocking streaming audio directly
16mServing Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing
33mTranslate tech talks using open-source models and cloud-native Wasm
41mCreating Versatile AI Agents Through Wasm+Rust
31mOrchestrating Multimodal RAG With Agentic Workflows