Open Source Model Performance Optimization With SGLang
summary
SGLang presents concrete performance wins: flash attention integration, dynamic MHA switching, kernel fusion and DP attention deliver ~2x speed on DeepSE F3; PD disaggregation and large-scale EP hit 52K input TPS per node, 22K output TPS per node and claim ~5x lower cost versus DSC API; a high-radix cache plus GPU-assisted IO yields up to 6x throughput and 84% lower TTFT; deterministic inference with cuda graph achieves 2.8x speedup; multi‑hardware support (NVIDIA, AMD, TPU, CPU) and OM Kubernetes operator enable one‑click Kim K2 deployments.
AI-written summary · about
chapters (10)
- 0:00Introduction and roadmap
- 0:512025 development highlights
- 1:16Technical deep dive
- 6:21Future outlook for 2025
- 7:09Community and Q&A session
- 11:42Training and deployment discussions
- 13:43In-depth optimization review
- 17:11Scaling and deployment architecture
- 21:39Model architecture support
- 22:53Final Q&A and conclusion
related talks
20mSGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving
44mIntroduction to LLM serving with SGLang
26mEfficiently serving LLMs at scale.
36mAccelerating High-Performance Machine Learning at Scale i...
15mScaling the Next Paradigm of Heterogeneous Intelligence
19mDeepSpeed – Efficient Training Scalability for Deep Learning Models
16mRevisiting Pipeline Parallelism for LLM Serving
19mOptimizations and abstractions for sparse machine learning