Preview: ML Training
summary
Survey of ML training papers summarizes concrete wins and techniques: the compute curve reaches 8 petaflop‑years for the largest GPT‑3 model; zero offload shows training models with trillions of parameters on hundreds of GPUs by spilling state to host NVM; Pollux adapts batch size and accelerator count to cut average job completion time by up to 50%; Refurbish Your Training Data reuses CPU preprocessing; ZICO schedules concurrent jobs to keep accelerator memory peaks low; Habitat predicts runtimes across accelerator types; Word and Octo enable on‑device and federated training on tiny, quantized hardware.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:24Model training in datacenters
- 2:04Overview of work covered in this talk
- 2:43Background: model training
- 3:47Activations, gradients, and weights too large
- 4:45Preprocessing can be a computational bottleneck
- 5:26How do multiple jobs share the same accelerator?
- 7:11How do we incorporate elasticity into schedulers?
- 8:16How about other kinds of objectives?
- 9:11How do we pick between accelerator types?
- 10:02Training in the datacenter not always attractive!
- 10:44Training on edge devices can be challenging!
- 11:14Can we train directly on edge devices?
- 11:42ML Training @ OSDI and ATC 2021
related talks
32mTaming the Machine: Basics of ML Models Training and Inference Optimization
36mAccelerating Machine Learning at Scale with
27mTips and tricks for distributed large model training
36mAccelerating High-Performance Machine Learning at Scale i...
41mGPU Training Made Easy-Pre-train & fine-tune NLP models
42mPost-Training Small Language Models: the adventures of a practitioner
01h 38mTraining Large Language Models: Practices and Research Questions
13mFrom Silicon to Scale: Training at Scale