WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
summary
Analysis finds up to 1.4× GPU compute skew in Llama 3 400B traces and pins sources to pipeline-level packing and context-level sharding. The system implements variable-length packing with outlier delay (20 ms overhead per batch) and fine-grained per-document sharding with an on-device kernel latency estimator to pick the fastest shard size. Evaluated on 500M–70B models, this balances attention and linear operator costs and yields about 1.23× speedup on average, larger gains at long contexts.
AI-written summary · about
chapters (7)
- 0:00WLB-LLM introduction
- 1:464D parallelism background
- 4:16Workload imbalance analysis
- 7:25Input packing imbalance solutions
- 10:59Document sharding and selection
- 14:10Performance improvements
- 15:50Q&A
related talks
15mResource Multiplexing in Tuning and Serving Large Language Models
31mThe Evolution of Multi-GPU Inference in vLLM
19mNo GPU Left Behind: Scaling Online LLM Training With...
19mAccelerating the Training of Large Language Models using Efficient Activation...
14mScalable Solutions for Running Large Language Models
16mRevisiting Pipeline Parallelism for LLM Serving
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference