SmartMoE: Efficiently Training Sparsely-Activated Models through Combiningand...
summary
SmartMoE is an automatic system that combines two-stage parallelization (offline pool construction + online selection) with a data-sensitive performance model to train MoE models faster. It adds expert placement as a hybrid-parallelism knob, builds a small candidate pool offline, and dynamically switches placement per iteration. Evaluation on three clusters up to 64 GPUs shows up to 1.88× end-to-end speedup over FasterMoE and an additional up to 1.4× gain from dynamic placement; the system and code are open source.
AI-written summary · about
chapters (12)
- 0:00SmartMoE introduction
- 1:20Scaling large models challenges
- 2:24Hybrid parallelism limits
- 3:50MoE model properties
- 5:39Why existing approaches fail
- 6:22SmartMoE two-stage parallelization
- 7:27Hybrid parallelism in SmartMoE
- 8:10Expert placement strategy
- 9:14Experimental results
- 10:17Ablation study validation
- 10:59Dynamic placement frequency
- 11:21SmartMoE contributions
related talks
18mSparse Checkpointing for Fast and Reliable MoE Training
37mUnstructured Sparsity Meets Tensor Cores: Lessons from Sparse Attention and MoE
14mSYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling
31mVeRL: Extreme Optimization Practices for Ultra-Large-Scale MoE Models
46mFlexOlmo: Open Language Models for Flexible Data Use
27mStageML: Partial Evaluation for Multi-Tenant MoE-LoRA Inference
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
46mSpeeding Up LLM Training by Over 45%