Train billion-parameter models with model parallelism on Amazon SageMaker
summary
Shows how SageMaker Model Parallel uses MPI, automated graph/module partitioning and pipeline parallelism with micro-batches to fit and train billion-parameter models that otherwise OOM. Examples: GPT‑2 XL (1.5B) trains on TensorFlow across p3dn instances; T5 (3B) scales on PyTorch to eight p3dn nodes. Usage details include a one‑process‑per‑GPU mapping, wrapping models/optimizers with distributed wrappers, decorating compute with SMP.step, and optional Horovod or DDP integration.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 1:14Challenges of large models
- 5:16Technical hurdles
- 10:14SageMaker library overview
- 12:35Pipeline training mechanism
- 13:59Using library with TensorFlow
- 22:20Using library with PyTorch
- 26:27Results and conclusion
related talks
25mAccelerate NLP training with Amazon SageMaker
39mTrain ML models at scale with Amazon SageMaker Training
21mCost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...
27mScaling ML to the next level: Hosting thousands of models on SageMaker
19mDeveloping large-scale machine learning algorithms on Amazon SageMaker
27mTips and tricks for distributed large model training
44mScaling ML from 0 to millions of users
13mLightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...