Why TTS Models Now Look Like LLMs
summary
Claims that modern TTS adopts an autoregressive transformer backbone applied to frame tokens and that neural audio codecs solve the information-density gap are demonstrated with concrete numbers and demos: open-weight TTS model, raw audio ≈200 kbps reduced by codecs to a few kbps, 80 ms frames at 12 fps represented as 37 tokens per frame ≈ 500 tokens per second. A live pipeline shows voice cloning from a few seconds and first-play latency of about 17 ms to first audio on one GPU; generation uses a diffusion-style frame decoder and streaming text input remains unresolved.
AI-written summary · about
chapters (17)
- 0:00Introduction and Mistral's new open-source TTS model
- 2:06Text-to-speech in AI agents and latency
- 3:33Live demo: Voice cloning with 'Paul'
- 6:00Voice cloning capabilities and multilingual examples
- 8:01Historical context of audio generation
- 8:55Transformer-based architecture for TTS
- 10:00Challenges of information density in audio
- 10:55Comparison of bit rates: text vs. audio
- 11:39Using neural audio codecs
- 13:10Backbone transformer and frame-based generation
- 14:56Text conditioning and model architecture
- 16:08Latency performance metrics
- 16:22Future outlook: Streaming text input
- 17:35Q&A: Generating text and audio simultaneously
- 18:24Q&A: Availability of voice cloning features
- 19:35Q&A: Philosophical take on speech interfaces
- 20:44Q&A: Next steps for streaming audio and text input
related talks
15mOptimizing inference for voice models in production
27mWhat happens when AI starts grocking streaming audio directly
20mMultimodal language models
17mServing Voice AI at Scale
14mLightning Talk: Enabling the Audio Modality for Language Models
33mTranslate tech talks using open-source models and cloud-native Wasm
22mBuilding next-gen speech synthesis for Bookmate audiobook service
21mCan AI Disrupt Speech Compression?