Spotnik: Designing Distributed Machine Learning for Transient Cloud Resources
summary
SPOTNIK implements an adaptive all-reduce ring that repairs membership without global checkpoints, uses an atomic barrier so partially applied parameter updates are discarded and retried, and can switch sync modes (SSGD vs ADP-SGD) based on cluster size (example policy: SSGD ≤12 workers, ADP-SGD >12). Recovery from VM revocation completes in sub-300 ms for ResNet/ImageNet in their experiments, and ring-repair messaging costs scale O(n·k) for n workers and k simultaneous revocations.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:21Distributed ML Train a machine learning model
- 1:01Challenges of distributed ML
- 1:59Transient cloud resources
- 3:06Implications of transient resources
- 4:36Current approach: Checkpoint & recovery
- 5:25Current approaches: Hybrid
- 6:03Spotnik
- 6:55Revocation recovery algorithm
- 9:11Atomic worker state update
- 10:17Adaptive synchronisation strategies
- 11:39Evaluation
- 14:23Conclusion
related talks
16mOptiReduce: Resilient and Tail-Optimal AllReduce for Distributed Deep Learning in the...
11mUnlocking the Power of Machine Learning for Cloud Workload Opt.
08mMaximizing ROI Through Distributed Computing and Multi-Cloud
48mResiliency and Availability Design Patterns for the Cloud
11mScaling Distributed Machine Learning with In-Network Aggregation
27mWhat I Wish I Knew Before Choosing Spot Instances
21mLessons Learned Running GKE Clusters on Spot Instances
17mCheckmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication