How Rust helped us building and deploying Moshi, a state-of-the-art speech model
summary
Moshi is a real‑time speech‑to‑speech system that ships an end‑to‑end inference stack in Rust, swaps PyTorch inference for Rust/Candle inference, and serves 500'000 Moshi sessions last summer on a low‑cost GPU cluster. The team moves tokenization and preprocessing into multi‑threaded Rust, hits ~200 ms end‑to‑end latency with 160 ms theoretical latency, and compresses audio to about 1 kilobit per second so browser tokenization plus quantized models make low‑bandwidth, deterministic serving feasible.
AI-written summary · about
chapters (11)
- 0:00QAI and Moshi intro
- 3:58Speech-to-speech model
- 7:52Voice assistant limits
- 13:48Natural speech generation
- 15:34Real-time Rust inference
- 21:16PyTorch Rust deployment
- 25:32Candle ML framework
- 35:57Scaling Moshi in cloud
- 45:31Rust for speech training
- 48:22Recent developments
- 50:39Q&A
related talks
21mMultimodal Open Source at Kyutai, From Online Demos to On-Device
30mOrchestrating Real-Time Multimodal AI Agents with Rust
04mOxidize Lightning Talks – Friedrich Rockenbauer
01h 00mFrom Tiny Chips to Server Racks: Deploying AI Everywhere with Rust
57mMoly - A pure Rust Explorer for Open Source LLMs
46mAdopting Rust by gracefully oxidising web applications
30mEnabling New Web Experiences
20mMastering Rust on Edge and Cloud — AI/IoT Case Studies