Revisiting Pipeline Parallelism for LLM Serving
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
Revisits pipeline parallelism with an implementation on SGLang that runs on four NVIDIA A100 40GB GPUs and evaluates Qwen2.5 32B and 14B. Introduces greedy and predictive chunk-sizing to adapt micro-batch size by predicted TFT/DPT slack and a delay scheduling that suspends and merges micro-batches to avoid 128-token tiling spikes. Shows pipeline PP with these mechanisms exceeds tensor parallel throughput, reaches ~95% goodput, and cuts decode-induced latency by ~31% and DPT by ~35%.
AI-written summary · about
chapters (12)
- 0:00Pipeline parallelism intro
- 1:24Prefill vs decode phases
- 2:30Throughput and goodput metrics
- 3:57Why underperforms in practice
- 4:40Pipeline bubbles from variable lengths
- 6:08Dynamic chunk prefetch
- 9:41Predictive chunk scheduling
- 11:07Delay scheduling for balance
- 12:53Evaluation setup
- 13:36ECP performance gains
- 14:42Delay scheduling benefits
- 15:25Summary
related talks
20mSGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving
22mCustom Pipeline Parallelism for Frontier-Scale Model Serving
44mIntroduction to LLM serving with SGLang
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
26mEfficiently serving LLMs at scale.
10mGraph Based Pipeline Parallelism
23mOperationalizing Large Language Model Training Pipelines