How to speed up any training on GPU
summary
Describes practical techniques Yandex uses to speed up machine learning training on large GPU clusters, covering both per-job and multi-job optimization. Mentions methods for making diverse training workloads run optimally across many GPUs and gives actionable ways to reproduce those improvements.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:51Why speed up training
- 2:33How GPUs work
- 4:20GPU utilization metrics
- 6:26Logs and reproducibility
- 7:11Training preparation
- 8:34Checkpoints and data loading
- 9:36Optimizing validation
- 10:40Increasing batch size
- 11:46Activation checkpoints
- 12:51GPU profiling capture
- 14:38GPU profiling analysis
- 16:23GPU network issues
- 16:45Parallelism types
- 20:24Zero parallelism
- 23:34Recommendations
- 25:46Summary
- 26:29Q&A
related talks
26mDeepSpeed: Efficient Training Scalability for Deep Learning
19mML training acceleration with heterogeneous resources in Bytedance
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
19mDeepSpeed – Efficient Training Scalability for Deep Learning Models
38mHow to run Neural Nets on GPUs'
13mPreview: ML Training
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM