CheckFreq: Frequent, Fine-Grained DNN Checkpointing
summary
CheckFreq implements two-phase checkpointing (snapshot in GPU or CPU then asynchronous persist) plus a resumable iterator and online profiling with adaptive rate tuning, integrated into PyTorch and NVIDIA DALI. It chooses k iterations (example: 15) to bound checkpoint cost, uses GPU snapshots that are ~10× cheaper than GPU→CPU copies, and pipelines snapshot/persist with computation. Experiments on ImageNet models and large pretraining show recovery drops from hours to seconds while keeping overhead to ≈3.5% (baseline 17–73%).
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:54DNN Training
- 2:14State of DNN Checkpointing Today
- 3:08Challenges for fine-grained checkpointing
- 4:47CheckFreq Design
- 5:592-Phase Checkpointing
- 6:23Example
- 8:03GPU-optimized Snapshots
- 8:40Checkpointing policy
- 8:55Systematic Online Profiling
- 9:27Outline
- 9:50Models and Experiments
- 10:14CheckFreq reduces checkpoint stalls
- 11:03Overall Training Overhead
- 11:26CheckFreq lowers recovery time
- 12:00Conclusion
related talks
16mCheck-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models
15mUniversal Checkpointing: A Flexible and Efficient Distributed Checkpointing...
37mTransparent, Infra-Level Checkpoint and Restore for Resilient AI/ML Workloads
13mByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development
17mCheckmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication
15mAdaCheck: An Adaptive Checkpointing System for Efficient LLM Training with Redundancy...
15mDistributed Checkpoint
18mSparse Checkpointing for Fast and Reliable MoE Training