Introduction to LLM serving with SGLang
summary
SGLang is an open-source serving framework that claims trillions of tokens per day in production and shows day‑zero support for models like DeepSeek and Qwen; deployment uses truss + a simple sglang server command on GPUs (L4/H100) with FP8, and tuning is flag-driven (notable knobs: CUDA graph max batch size and Eagle 3 speculative decoding). Benchmarks use LM‑eval/LM4, raising max CUDA graph from 8 to 32 makes decode CUDA graphs effective and lifts per‑user throughput from ~15 to ~12+ tokens/sec acceptance rates depend on Eagle parameters; demos deploy, benchmark, and tune successfully on BaseTen GPUs.
AI-written summary · about
chapters (9)
- 0:00Introduction to LLM serving with SGLang
- 2:14What is SGLang?
- 3:36History of SGLang
- 6:49Deploying Your First Model
- 13:01Optimizing Performance with CUDA Graph Max Batch Size
- 24:19Optimizing Performance with Eagle 3 Speculative Decoding
- 30:02SGLang Community and Contributions
- 35:24Invitations and Job Opportunities
- 36:52Q&A
related talks
20mSGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving
25mOpen Source Model Performance Optimization With SGLang
16mRevisiting Pipeline Parallelism for LLM Serving
28mOptimizing Long-Tail and MoE Challenges in Reinforcement Learning with SGLang
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
26mEfficiently serving LLMs at scale.
10mRunning LLMs locally: Practical LLM Performance on DGX Spark
15mAI Trends: Insights