Sparse Checkpointing for Fast and Reliable MoE Training
summary
MoEvement is an in-memory checkpointing system for MoE models that implements sparse checkpointing, sparse-to-dense conversion, and upstream logging to avoid stalls and localize recovery. Built on DeepSpeed and evaluated on 96 A100s with a 16.4B DeepSeek model (64 experts, 2.8B active), it reduces snapshot size ~4×, cuts recovery overhead up to 31× (sustains ETTR ≥0.94 down to MTBF = 10 minutes), and delivers up to 8× overall training speedup while preserving synchronous semantics.
AI-written summary · about
chapters (7)
- 0:00Movement checkpointing intro
- 2:50Checkpointing MoE challenges
- 4:58Trade-off in checkpointing
- 8:56Recovering from sparse checkpoints
- 13:32Evaluation setup
- 16:25Performance gains results
- 17:50Summary
related talks
12mSmartMoE: Efficiently Training Sparsely-Activated Models through Combiningand...
15mUniversal Checkpointing: A Flexible and Efficient Distributed Checkpointing...
22mOptimizing Large MoE Inference on NVIDIA Blackwell: NVFP4, ADP, and DualPipe Strat...
20mAccelerating Distributed MoE Training and Inference
13mByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development
16mCheck-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models
37mUnstructured Sparsity Meets Tensor Cores: Lessons from Sparse Attention and MoE
19mDeepSpeed – Efficient Training Scalability for Deep Learning Models