Lifecycle of a Parameter
summary
Follows one parameter through an LLM RL post‑training pipeline: reading 100s of GB checkpoints, doing FP8→BF16 dequant on load, splitting the tensor across four GPUs as TP4 + DP4 sharding, updating shards, reconverting and resharding for a different inference layout, and saving a distributed checkpoint. Compares direct GPU transfer vs staging to CPU RAM and asynchronous vs synchronous sync (staleness vs pause time). Torch Forge prototypes these mechanics and demonstrates the tradeoffs and latency/memory numbers.
AI-written summary · about
chapters (12)
- 0:00Parameter lifecycle at scale
- 3:32Simple models to trillion-parameter clusters
- 5:19Parameter representation and download
- 7:24Loading parameters into GPUs
- 9:49Sharding strategies for large models
- 15:10Dequantization and precision handling
- 16:53Training updates and sync
- 19:00Reversing for inference
- 21:06Sync methods compared
- 22:30Saving final checkpoint
- 23:32Parameter lifecycle complexity
- 24:57Future challenges and closing
related talks
30mUnderstanding the end-to-end LLM training and inference pipeline
43mState of GPT
34mPanel Discussion: Training and Inference at Planet Scale
23mOperationalizing Large Language Model Training Pipelines
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
21mLessons Learned in WatsonX Training: Scaling Cloud-Native...
16mRevisiting Pipeline Parallelism for LLM Serving
21mWhy Number of Parameters Isn't a Good Performance Indicator for Live AI Processing