Metis: Fast Automatic Distributed Training on Heterogeneous GPUs
summary
Metis automatically finds parallelism plans for training across mixed GPU types by profiling each GPU, estimating cost, and using a planner that prunes skewed device-group mappings, balances layer assignment by relative stage compute power, and prefers data parallelism over tensor parallelism within stages. Evaluation on GPT-3, MoE, and Wide-ResNet across P100/T4/V100 mixes shows Metis yields 1.05–8.43× speed-up, matches an oracle within ~5% while cutting profiling/search time by orders of magnitude (100× vs. Alpa).
AI-written summary · about
chapters (7)
- 0:00Metis introduction and goals
- 2:12Limitations of auto-parallelizers
- 4:00Why abstraction fails
- 7:10Metis system overview
- 8:36Core planner techniques
- 14:54Experimental results
- 18:47Summary and future
related talks
26mEfficient MoE Pre-training at Scale on AMD GPUs With TorchTitan
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
07mDemo: Neural Network Acceleration on Metis, Powered by RISC-V
21mKTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for M...
37mDistributed Deep Learning on Apache Mesos with GPUs and Gang Scheduling
26mDeepSpeed: Efficient Training Scalability for Deep Learning
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
15mAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving