AdaCheck: An Adaptive Checkpointing System for Efficient LLM Training with Redundancy...
summary
AdaCheck is an adaptive checkpointing system that minimizes checkpoint size by identifying tensor redundancy across parallelisms, model architectures, and iterations, and it implements a hash-based detector with a ring-based communication algorithm to find duplicate tensors. It adds a gradient-based incremental checkpointing mode that offloads gradients and CPU-side optimizer state, and the prototype reduces checkpoint size by 6–896× and increases checkpointing frequency by 1.46–111× with almost no training-throughput overhead.
AI-written summary · about
chapters (8)
- 0:00AdaCheck introduction
- 2:08Existing checkpoint limits
- 4:14AdaCheck core challenges
- 7:06Offline redundancy method
- 9:39Tensor redundancy detector
- 12:36Online redundancy method
- 13:18Experimental results
- 14:26Summary and future
related talks
15mUniversal Checkpointing: A Flexible and Efficient Distributed Checkpointing...
16mCheck-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models
17mCheckmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication
10mLightning Talk: In-Cluster Distributed Checkpointing: Optimizing Training...
12mCheckFreq: Frequent, Fine-Grained DNN Checkpointing
15mDistributed Checkpoint
33mBeyond Loss and Accuracy: Closing the Observability Gaps in AI Training with...
13mByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development