SGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving
summary
SGLang is an open source LLM inference engine that implements PD disaggregation and hierarchical KV cache offloading, adds large-scale expert parallelism and speculative decoding, and reproduces DeepSeek results at scale. Benchmarks claim ~3× faster preview throughput, 5.1× decoder throughput, and cost parity around $0.20 per 1M tokens; deployments include xAI, NVIDIA and AMD. Design details cover prefill/decode pairing, RDMA/non-blocking KV transfer, a load balancer that selects PD pairs, launch-order fixes for two-batch overlap, and an EPLB rebalance that improves imbalance across hundreds of contributors.
AI-written summary · about
chapters (6)
- 0:00Introduction to SGLang
- 1:13Milestones and Features
- 3:30PD Disaggregation Design
- 7:38Expert Parallelism and V3
- 15:44Load Balancing Challenges
- 17:32Ecosystem and Future Work
related talks
44mIntroduction to LLM serving with SGLang
25mOpen Source Model Performance Optimization With SGLang
14mScalable Solutions for Running Large Language Models
16mRevisiting Pipeline Parallelism for LLM Serving
24mLLM Inference Optimization From Simple To Sophisticated
15mDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
32mEnd-to-End Optimizing Multi-Turn RL and High-Performance Inference in Agents with...