Optimizing inference for voice models in production
summary
Demo uses Orpheus TTS (Llama 3.2 3B backbone) served with TensorRT-LLM + FP8 quantization and torch.compile on GPU for audio decoding, achieving ~150 ms TTFB and 16–24 simultaneous streams on an H100 MIG (half GPU). Runtime optimizations include KV-cache FP8, dynamic batching with 15 ms flushes, and GPU-side decoding with SNACK, but real bottlenecks are CPU-bound loops, HTTP session overhead, DNS hairpins and client-side sequential requests which can erase gains unless websocket/gRPC and shared sessions are used.
AI-written summary · about
chapters (8)
- 0:00Introduction and agenda
- 1:04TTS model architecture overview
- 3:30Performance metrics for voice models
- 5:15Optimization techniques for inference
- 8:28Benchmarking and results
- 10:35Pitfalls in infrastructure and client code
- 13:04Voice agent pipeline considerations
- 14:28Summary and Q&A
related talks
22mWhy TTS Models Now Look Like LLMs
33mDesigning Voice Agents for Real Conversations
43mSpeeding up Whisper for speech recognition by 10x - Optimisations and walkthrough
35mTricks Learned from Scaling WhisperSpeech Models to 80k+ Hours of Speech
12mDemo: Optimizing Gemma inference on NVIDIA GPUs with TensorRT-LLM
25mEngineering voice agents: Latency, quality, and scale
18mDeploying ML solutions with low latency in Python
27mWhat happens when AI starts grocking streaming audio directly