Tips and tricks for distributed large model training
summary
Overview of practical distributed training strategies with concrete tooling and results: shows training a 39-billion-parameter language model on Cloud TPUs using XLA SPMD and the T5X framework, demonstrates pipeline (GPipe) and tensor (Mesh TensorFlow style) sharding with AllGather/ReduceScatter communication, explains rematerialization vs offload trade-offs and device mesh specs (example TPU VM 3×8 setup with Jupyter), and reports high measured vs effective FLOP utilizations across TPU v4 clusters when iterating sharding configs.
AI-written summary · about
chapters (7)
- 0:00Start
- 1:16Data Parallelism
- 6:22Pipeline Parallel
- 6:47Tensor Parallel
- 7:19Model Parallelism Approaches
- 9:16Spatial Partitioning
- 17:15Compute and Communication Overlap
related talks
24mTraining large scale models using PyTorch
14mLightning Talk: Large-Scale Distributed Training with Dynamo and...
01h 03mScaling up machine learning without tears (and what do programming languages ha...
28mSupercharge your training on TPUs
08mPyTorch Distributed: Towards Large Scale Training
43mDeep learning beyond the learning
29mTrain billion-parameter models with model parallelism on Amazon SageMaker
28mTraining Large Language Models on Kubernetes