Lightning Talk: In-Cluster Distributed Checkpointing: Optimizing Training...
summary
PyTorch Distributed Checkpointing adds an In-Cluster Checkpointing mode that saves and restores each GPU node’s state to node-local storage, replicates those local checkpoints to replacements on recovery, and falls back to persistent storage periodically. Engineering includes pin-memory staging with forward-step overlap, rank-local checkpointing to avoid global collectives, and process-based checkpointing to eliminate GIL contention. At scale this reduces checkpoint overhead to near zero, claims show up to 5% better goodput and >50% reduction in checkpoint-related badput, saving thousands of GPU hours.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:23Checkpointing overheads
- 2:49Staging memory bottlenecks
- 3:51Optimizations for staging
- 4:52Measuring efficiency
- 6:15Checkpointing impact on training
- 7:18In-cluster checkpointing solution
- 8:00How in-cluster checkpointing works
- 9:01Experimental results
- 10:01Q&A
related talks
17mCheckmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication
15mUniversal Checkpointing: A Flexible and Efficient Distributed Checkpointing...
15mDistributed Checkpoint
08mPyTorch Distributed: Towards Large Scale Training
25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...
15mAdaCheck: An Adaptive Checkpointing System for Efficient LLM Training with Redundancy...
19mComposable Distributed PT2(D)
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads