How Open-Source vLLM Topped the Artificial Analysis Leaderboard
summary
Open-source vLLM plus DigitalOcean infra delivers concrete wins: B100 GPUs with NVB4 quantization and tensor sharding cut memory and speed bottlenecks, kernel fusion reduces attention-layer kernel launches from 33 to 10 for a ~1.2× speedup, and overall TPT drops from ~7–8 ms to ~4 ms yielding 230 tokens per second on DeepSeek v3.2 and sub‑second time‑to‑first‑token for 10,000‑token Qwen 3.5 prompts; speculative decoding with an EAGLE3 raft model and overlapped kernel launches (PDL) improve decode throughput, and all fixes are upstreamed into vLLM.
AI-written summary · about
chapters (8)
- 0:00Introduction and context
- 1:11GPU selection and benefits
- 4:07Model quantization and NVB4
- 5:32vLLM optimizations overview
- 6:34Kernel fusion deep dive
- 10:26Leaderboard results and metrics
- 11:30Speculative decoding and PDL
- 12:11Open-source impact and future
related talks
35mThe State of vLLM
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
39mAccelerating Open-Source RL and Agentic Inference with vLLM
24mLLM Inference Optimization From Simple To Sophisticated
21mScaling LLM Inference with vLLM
16mHow fast are LLM inference engines anyway?
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference