SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling
summary
SYMI claims a practical solution: it decouple optimizer placement from expert parameters so optimizer state is static optimizer sharding across nodes while expert weights move with no extra data transfer by repurposing weight-update communication. It supports rebalance every iteration, dropping far fewer tokens, needs ~29% fewer iterations versus DeepSpeed, and achieves 30.5% and 25.9% faster time-to-convergence than DeepSpeed and FlexMoE respectively; optimizer state can be 8–32× larger than weights and SYMI adds only ~1% locality overhead while keeping latency close to static placement.
AI-written summary · about
chapters (13)
- 0:00MoE training introduction
- 1:01How mixture of experts architecture works
- 2:25MoE deployment challenges
- 3:51Convergence vs latency trade-off
- 4:55Adaptive expert replication
- 6:00Optimizer state overhead
- 7:25SimMi design and decoupling
- 8:49SimMi workflow per iteration
- 10:18Evaluation setup and baselines
- 11:01End-to-end training time
- 12:26Convergence improvements with SimMi
- 13:09Latency comparison
- 13:52Conclusion and Q&A
related talks
12mSmartMoE: Efficiently Training Sparsely-Activated Models through Combiningand...
31mVeRL: Extreme Optimization Practices for Ultra-Large-Scale MoE Models
46mSpeeding Up LLM Training by Over 45%
45mSymbiotic Relations between Decoupled Training, Optimization, and Federated Learning
37mUnstructured Sparsity Meets Tensor Cores: Lessons from Sparse Attention and MoE
47mMeta-learning of Optimizers and Update Rules
46mFlexOlmo: Open Language Models for Flexible Data Use
17mSparseZETA: Intelligent Auto-tuner for Designing High-Performance SpMV Programs