Accelerated LLM Inference with Anyscale
summary
Anyscale details concrete LLM inference gains: contributions to vLLM add FP8 support (benchmarks show ~1.27x TTF and 1.5x token latency on H100), chunked prefill (up to 1.4x ITL at 30% TTF cost), and speculative decoding (typical 1.5–2x speedups); custom fused kernels yield 15–30% faster decoding, fast model loading cuts cold-start time 4–5x so replicas come up in ~2 minutes, and Ray AIR batch reports ~3x cost savings (up to ~6x with prompt prefix sharing).
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:25Open source contributions
- 2:00FP8 support deep dive
- 3:58Chunked prefill
- 5:40Speculative decoding
- 9:38Proprietary inference stack
- 12:08RayLLM serving solutions
- 13:03RayLLM batch inference
- 16:05JSON mode and T3 algorithm
- 18:54Fast model loading
- 20:32Upcoming preview features
- 23:02Summary and takeaways
- 23:58Q&A session
related talks
24mLLM Inference Optimization From Simple To Sophisticated
28mSponsored Session: Arctic Inference: Breaking the Speed-Cost Tradeoff in LLM Serving
38mOptimizing LLM Inference for the Rest of Us
28mArctic Inference: Open-Source LLM Inference Optimizations at Snowflake
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
21mScaling LLM Inference with vLLM
11mWhen Less Is More: A Technical Overview of LLMs and the Strength of Smaller Models
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone