From Silicon to Scale: Training at Scale
summary
AMD shows a concrete stack and outcomes: full-stack hardware/software co‑design using rocm, PyTorch, Triton and Hugging Face, hybrid parallelism (tensor + pipeline + data + spatial) that attains 113B parameters with 91 climate variables and 1.6×FLOPS on 49,000 GPUs, 100% weak scaling and up to 89% strong scaling for 175B–1T models, and a 3,000‑GPU recipe for GPT‑scale work. Graph NN experiments hit 90% cluster utilization on 16,000 GPUs for 100M graphs using 400 hyperparameters pruned to the top 20 configs, and fused comm/compute cuts runtime ~30%.
AI-written summary · about
chapters (9)
- 0:00AMD scaling efforts
- 1:09Hardware-software co-design
- 2:58Open-source collaborations
- 3:21Efficient language model training
- 4:46Climate modeling with hybrid parallelism
- 8:02Material science with graph neural networks
- 10:55Key techniques and impact
- 11:18Overcoming communication bottlenecks
- 12:43AI scaling roadmap
related talks
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
12mKeynote: AI Infrastructure
34mDeep Learning at Scale
21mDesigning Scalable Networks for Large AI Clusters: Challenges and Key Insights
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
26mEfficient MoE Pre-training at Scale on AMD GPUs With TorchTitan
28mScaling AI Infrastructure
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective