Diffusion to Prod: vLLM-Omni Unifies Multimodal Inference
summary
A single runtime unifies autoregressive and diffusion generation so one pipeline handles text, image, audio and video rather than two stacks, implemented as vLLM-Omni with over 40 model architectures supported. Stage placement uses YAML so Thinker→Talker→Vocoder replicas scale independently and cross-node hops use Mooncake/RDMA. Diffusion caching (TCache / CashDits) shows up to 4× speedups, TTS streaming achieves ~12× faster time‑to‑first‑audio, and a recorded Cosmos 3 demo runs on an H100. Quantization to FP8/FP4 and V22 release shipping.
AI-written summary · about
chapters (20)
- 0:00AI Generated Intro
- 0:40Speaker Background Resources
- 1:57Talk Agenda Overview
- 3:42Why Multimodal Breaks Stacks
- 5:17One Engine Two Brains
- 6:28Model Support Ecosystem
- 7:38AR vs Diffusion Deep Dive
- 9:32API Surface And Frontends
- 11:29Runtime Components Tour
- 13:37Omni Connector Data Flow
- 17:22Pipeline Stages Example
- 19:51Deploy YAML Placement
- 23:16Serving Topologies Repo Layout
- 26:35Diffusion Fundamentals Primer
- 29:45Acceleration Caching Parallelism
- 33:35Multi Node Deployment With Mooncake
- 34:40Cosmos 3 Demo Replay
- 39:38Streaming TTS Benchmarks
- 41:03Release Roadmap Wrap Up
- 43:55Links Final Thanks
related talks
28mText Diffusion
39mDemocratizing Diffusion Models with Diffusers
20mRealtime Video AI with Diffusion Models
50mAI'll Be Back: Exploring Diffusion Models for Image, Audio & Video
33mTranslate tech talks using open-source models and cloud-native Wasm
19mYou Might Not Need 50 Diffusion Steps
40mDiffusion Large Language Models
22mWhy TTS Models Now Look Like LLMs