MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
Talkat 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24) · 2024-04-16..04-18·
summary
MegaScale is a production system that trains LLMs on more than 10,000 GPUs and achieves concrete gains: 55.2% Model FLOPs Utilization when training a 175B model on 12,288 GPUs, a 1.34x MFU improvement over Megatron‑LM and up to 1.35x end‑to‑end speedup enabling 175B over 300B tokens in 1.75 days. Key engineering steps include overlapping communication with computation in 3D parallelism, a two‑stage checkpointing that lets GPUs resume immediately, and reducing initialization from hours to 30 seconds.
AI-written summary · about
chapters (6)
- 0:00Introduction to MegaScale
- 1:15Scaling challenges
- 4:05Efficiency techniques
- 9:12Robust training framework
- 12:37Deployment and performance
- 15:41Conclusion and future work
related talks
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
23mOperationalizing Large Language Model Training Pipelines
19mSystem for Scaling Large Language Models
31mBuilding Large-scale Localized Language Models
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
22mPerformance Optimizations at 100K+ Scale
13mFrom Silicon to Scale: Training at Scale