~/talk

Distributed Checkpoint

on YouTube 15m
Talkat PyTorch Conference 2023 · 2023-10-16..10-17· [distributed-systems, machine-learning]

speakers

name:Iris Zhang
links:none on file
name:Chien-Chin Huang
links:none on file

resources

video:https://youtu.be/ldBmHNva_Fw
slides:none on file

summary

Distributed checkpointing saves and loads model and optimizer shards in parallel, writes one file per rank plus a metadata file, and performs load-time resharding so a checkpoint saved on 8 GPUs can resume on 4 (or even without GPUs). It stores parallelism-agnostic tensors and requires a preprocessing helper (the distributed checkpoint state-dict utilities) to produce canonical keys and re-shardable values. Adoption includes IBM, Mosaic, XLA for FSDP and Shampoo OSS; APIs are beta in 2.1 stable with helpers experimental in nightly. save/load from multiple ranks

AI-written summary · about

chapters (7)

  1. 0:00Distributed checkpointing intro
  2. 1:48Changing world size challenges
  3. 4:18Three key rearing challenges
  4. 7:08PyTorch native solution
  5. 9:59Handling state values and keys
  6. 12:49State API for simplicity
  7. 14:59Q&A

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.