Transparent, Infra-Level Checkpoint and Restore for Resilient AI/ML Workloads
Talkat KubeCon + CloudNativeCon Europe 2025 · on Friday, 2025-04-04· at 14:30· Level 1 | Hall Entrance S10 | Room B·
summary
Demonstrates a Kubernetes operator that uses CRIU, CRI-O, a modified runc and cuda-checkpoint to transparently checkpoint and hot-restart distributed PyTorch jobs; two demos show a manual periodic checkpoint that rolls back to the last epoch and an automated JobSet migration that resumes on a spare node. Asynchronous checkpointing cuts interruption windows by 30–100x, compression reaches up to 5:1, and implementation uses a daemonset plus rsync to move checkpoint files; drawbacks include larger image-sized checkpoints, privileged runtime needs, and restore-machine constraints.
AI-written summary · about
chapters (9)
- 0:00Transparent checkpointing for AIML
- 2:08GPU resilience approaches
- 7:07Infra-level checkpointing defined
- 9:57GPU workload challenges
- 13:08Checkpointing trade-offs
- 16:43Infra-level checkpointing in Kubernetes
- 28:00Periodic checkpoint demo
- 32:57Job set migration with checkpointing
- 35:25Next steps and collaboration
related talks
11mLightning Talk: Enabling Hot Restart of Stateful Applications Including GPU-Accelerate...
39mKubernetes and Checkpoint/Restore
15mUniversal Checkpointing: A Flexible and Efficient Distributed Checkpointing...
12mCheckFreq: Frequent, Fine-Grained DNN Checkpointing
15mDistributed Checkpoint
16mCheck-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models
25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...
17mCheckmate: Zero Performance Overhead Model Checkpointing via Network Gradient Replication