Lessons Learned in WatsonX Training: Scaling Cloud-Native...
summary
Claims PyTorch FSDP and a hybrid HSDP/DDD approach scale model pretraining from 100M to 20B parameters using up to 512 GPUs, training on >10 TB of web data with sequence lengths up to 4K. Built and validated sharded checkpointing that cuts checkpoint time from hours to minutes and a stateful distributed dataloader that avoids rereading data across restarts. Solved stability problems by removing synchronous validation, mirroring prefix lengths for PLM, and documenting activation‑checkpointing tradeoffs; production runs currently stabilize around 7–13B with plans for larger models.
AI-written summary · about
chapters (13)
- 0:00WatsonX training scaling intro
- 1:02From 110M to 20B parameter models
- 2:26Scalable training design space
- 4:52Parallelization schemes: DDP, FSDP, hybrid
- 7:18Stability in large-scale training
- 9:50Checkpointing at scale
- 11:37Beyond compute and memory
- 13:43Stateful data loaders
- 14:48Activation checkpointing trade-offs
- 16:13CPU memory and streaming loaders
- 16:57CPU latency and Torch compile
- 17:3880-20 rule in training
- 18:19Q&A
related talks
24mTraining large scale models using PyTorch
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
34mDeep Learning at Scale
13mFrom Silicon to Scale: Training at Scale
27mTips and tricks for distributed large model training
08mPyTorch Distributed: Towards Large Scale Training
25mDistributed Training in PyTorch: Zero to Hero
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs