Everything Everywhere all at Once: vLLM...
summary
vLLM TPU backend now supports both PyTorch and JAX under one code path and ships a new Ragged Page Attention kernel v3 (Palace) and XLA lowering that yields 2–5x performance from prototype to launch. Trillium TPUs saturate at about 14k tokens/sec for a 1500-token prefill and cost roughly $0.48 per million tokens; H100s give higher raw throughput but higher cost. Real-time safety classification sustains ~4x the QPS under a 50 ms P99; decode‑heavy playlist workloads need ~2x more TPU capacity and showed a 62% cost reduction in Spotify’s benchmarks.
AI-written summary · about
chapters (19)
- 0:00vLLM and hardware interoperability
- 1:03What is vLLM and why it matters
- 2:29Spotify uses vLLM in production
- 3:12Demo of vLLM AI features at Spotify
- 7:50Optimizing vLLM for low-latency AI
- 9:18Why Spotify explored TPUs
- 9:39vLLM TPU backend launch
- 11:02Key changes in vLLM TPU backend
- 12:28Hardware interoperability in vLLM
- 14:58Spotify’s TPU production evaluation
- 15:40Benchmarking TPUs for throughput
- 18:35Real-time latency benchmarks
- 21:28TPUs versus GPUs at Spotify
- 22:33Future plans for TPUs and multimodal
- 23:13Introducing Ironwood TPU
- 24:40Ironwood focus areas for AI inference
- 25:24Kubernetes and LMD integration
- 25:44Distributed RL with vLLM
- 26:06Resources for vLLM and TPUs
related talks
42mCloud TPU Pods: AI Supercomputing for Large Machine Learning Problems
23mTorchTPU: Expanding TPU Programmabil...
15mAI Trends: Insights
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
16mDe-mystifying PyTorch for ASICs: When (and Why) To Move Your Development To AI A...
17mEnable Generative AI Everywhere with Ubiquitous Hardware and Open Software
15mAI Trends, and Running LLMs in the Enterprise