Multimodal Open Source at Kyutai, From Online Demos to On-Device
summary
Nonprofit lab open sources multimodal tools: full-duplex 7B Moshi (cloud on L4 or quantized on MacBook Pro), a neural audio codec Mimi to tokenise speech, and Moshi Vision +200M params trained with ~20,000 hours of synthesized conversation for low‑latency multimodal dialogue. Releases include a 24‑language EU LLM, a French→English speech‑to‑speech with voice cloning that runs real-time on recent iPhone, and demonstrations of >300 concurrent generations on one H100. Multiple implementations (PyTorch training, Rust inference, CoreML/Swift) and CUDA graph optimization fix latency; quantization formats remain fragmented.
AI-written summary · about
chapters (10)
- 0:00Kyutai and open science
- 1:49Why open source matters
- 4:17Neural audio codec Mimi
- 7:07Moshi vision
- 9:14Multilingual LLM
- 10:39WebPile and model soup
- 12:02ED live translation
- 13:49Maintaining models
- 17:23Inference with CUDA Graphs
- 20:36Conclusion
related talks
20mMultimodal language models
40mKeynote: Open-source Multimodal AI in the Wild
01h 01mHow Rust helped us building and deploying Moshi, a state-of-the-art speech model
39mLive voices with LiveView
42mdata.bythebay.io: Shubho Sengupta,
23mNeMo: Open Source Toolkit for Conversational AI
30mLook ma, no hands! Multimodal AI-Agents in the browser
23mBuilding with OpenAI: What's Ahead