Enabling Secure Container Checkpointing for Distributed Model Training
summary
Extends CRIU and container runtimes to synchronize checkpoint and restore across multiple containers and nodes using a new CUDA checkpoint + CRIU coordinator that orchestrates distributed PyTorch jobs and Kubernetes pods. Implements CRIU-integrated end-to-end encryption so memory pages and CRIU images are encrypted before hitting disk, avoiding extra read/write passes and outperforming gpg/openssl. Checkpoints complete in a few seconds, restores are faster, supports Docker Podman containerd and is being trialed with CubeFlow for hot swapping and maintenance.
AI-written summary · about
chapters (9)
- 0:00Secure container checkpointing intro
- 1:24Checkpointing for distributed training
- 2:50GPU state checkpointing challenges
- 3:57Synchronizing distributed checkpoints
- 4:38Securing checkpoints with encryption
- 6:48Encryption strategies by data type
- 7:12Distributed checkpoint restore demo
- 8:16Evaluation results and impact
- 9:17Future plans and collaboration
related talks
24mProtecting Sensitive Data in Container Checkpoints
32mEnd-to-End Encryption for Container Checkpointing in Kubernetes
33mFreeze,Investigate,Recover: Unlocking Forensic Container Checkpoint...
31mDigital Forensics with Container Checkpointing- 31mDigital Forensics with Container Checkpointing
37mCSI Forensics: Unraveling Kubernetes Crime Scenes
38mIntroduction to CRI-O
33mDon't Peek Into my Container!