Galvatron: An Automatic Distributed Training System for Efficient Large...
summary
Galvatron is a PyTorch-native framework that profiles a Transformer’s execution to build a PyTorch profiler cost model, then prunes a huge hybrid-parallel search space with decision-tree + dynamic programming search to pick per-layer mixes of data, tensor, pipeline, sharded-data, sequence parallelism and recomputation. Runtime uses FSDP, layer-granularity checkpoints and fused kernels to deploy with minimal code changes across NVIDIA GPUs and Ascend NPUs. Experiments report up to an 80% speedup over Megatron 1.1 on large-cluster configurations.
AI-written summary · about
chapters (15)
- 0:00Galvatron introduction
- 1:04Large model training challenges
- 2:30Need for auto parallelism
- 3:14System design goals
- 3:58Parallelism strategies
- 5:21Fine-grain hybrid parallelism
- 7:07Auto optimization with cost modeling
- 8:12System features and evaluation
- 8:35PyTorch-native details
- 10:01Automation engine
- 11:24Search engine and DP
- 12:28Hardware and model versatility
- 12:48User integration
- 13:30Experimental results
- 14:38Summary and future
related talks
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
26mThunder: Distribute and Optimize Your PyTorch Models With Zero...
29mEnabling Lightweight, High-Performance FSDP With NVIDIA GPU
37mTraining and Optimisation of Large Transformer Models: An ATLAS and CERN Use Case - Ricardo Rocha, CERN & Maxence Draguet, University of Oxford
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
45mTransformers for Game Log Data
24mKeynote: Next-generation frameworks for Large-scale AI
18mPetS: A Unified Framework for Parameter-Efficient Transformers Serving