Checkmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication
summary
Claims a system that achieves per-iteration checkpointing with zero GPU overhead by multicasting gradients off the training path and applying them on a CPU-based shadow cluster. Implementation uses a P4-capable switch and a nickel plugin to do multicast gradient replication with a selective replication algorithm that needs only twice the per-GPU bandwidth per collective group. Evaluation on A100 nodes shows throughput within 0.8% of a no-checkpoint baseline, 5–34.5× more frequent checkpoints, and 80–97.1% less repeated work.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:51Checkpointing trade-off
- 4:52Zero-overhead checkpointing
- 7:20Checkmate architecture
- 8:22Selective gradient replication
- 11:10Reliable gradient delivery
- 12:35Storage-side computation
- 13:37Performance with Llama 2 13B
- 15:20Scaling with storage nodes
- 16:22Summary
related talks
10mLightning Talk: In-Cluster Distributed Checkpointing: Optimizing Training...
15mUniversal Checkpointing: A Flexible and Efficient Distributed Checkpointing...
13mByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development
16mCheck-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models
15mDistributed Checkpoint
32mEfficient Transparent Checkpointing of AI/ML Workloads in Kub...
25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...
12mCheckFreq: Frequent, Fine-Grained DNN Checkpointing