At-scale Training
summary
Shows how SageMaker Model Parallel + PyTorch actually partitions and trains a GPT-J–class model across P3 and P4 instances, using pipeline, tensor and data parallelism plus optimizer state sharding and activation checkpointing/offloading. Demonstrates automatic graph tracing and partitioning, measures: initial P3 run traces out of memory then runs in four instances with ~42 minutes end to end, switching to P4d and fp16 plus EFA and tuned microbatches cuts wall time to about 10 minutes and reduces cross‑node communication.
AI-written summary · about
chapters (4)
related talks
23mSponsored Session: Powering PyTorch: Decentralized Training for an Energy-Hungry Future
19mBeat GPT 4 & Build the Best AI For You
19mDeveloping large-scale machine learning algorithms on Amazon SageMaker
46mWhen AI Meets High Performance Computing
23mPyTorch on Google Cloud TPUs
01h 38mAccelerate Model Hyperparameter Search with RAPIDS and SageMaker
58mMastering PrivateGPT: Tailoring GenAI for your unique applications
01h 00mHigh-performance & cost-effective model deployment with Amazon SageMaker