Understanding Stragglers in Large Model Training Using What-if Analysis
summary
Five-month ByteDance cluster traces, with jobs up to >5,000 GPUs, drive a counterfactual what-if simulation that replaces computation times with the mean and communication times with the median and replays dependencies via a topological sort to estimate ideal runtime. Results show ~42.5% of jobs suffer >10% slowdown (mean slowdown 1.15, tail up to 2x), worker-health is rarely the dominant cause, PP stage partition imbalance hits ~39.3% of jobs, and the analysis is deployed as ESMO monitoring service for real-time detection and heatmap diagnosis.
AI-written summary · about
chapters (8)
- 0:00Stragglers in model training
- 1:23What-if analysis approach
- 3:53Trace collection details
- 6:00Ideal job runtime
- 8:08Prevalence and impact
- 9:13Root causes identified
- 12:28Production monitoring use
- 15:01Summary and key findings
related talks
17mCharacterization of Large Language Model Development in the Datacenter
19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
18mGREYHOUND: Hunting Fail-Slows in Hybrid-Parallel Training at Scale
24mTaming Data Skew in Production ML Pipelines
15mMLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU...
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
21mExperience Productionizing and Operating GB200 Cluster
21mEffective Straggler Mitigation: Attack of the Clones