Taming the Machine: Basics of ML Models Training and Inference Optimization
summary
Covers concrete knobs and measured wins: mixed precision (≈3× speedup, sometimes 5×), batch-size + optimizer tuning, gradient accumulation and checkpointing to trade memory for scale, and ZeRO for multi‑GPU memory savings. Shows torch.compile (claimed ≈51% training speedup), ONNX Runtime and NVIDIA TensorRT for inference with INT8 quantization and quantization‑aware training, Triton for serving, and Mojo as a promising Python‑like path to C++ performance on Llama‑7B benchmarks.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:46Training versus inference overview
- 3:31Mixed precision training
- 4:56Batch size optimizer
- 6:40Gradient checkpointing
- 7:45Data preloading
- 8:27Compiler optimization
- 10:35TorchCompile
- 11:39Multi-GPU training
- 13:22ZeRO optimization
- 14:48Production ML code
- 16:35Inference optimization
- 17:18ONNX for sharing
- 18:24ONNX Runtime
- 19:27Quantization for edge
- 20:52TensorRT for NVIDIA
- 21:35Scalable ML serving
- 23:43Python limits and Mojo
- 25:51Mojo acceleration
- 27:20Q&A
- 29:49CPU versus GPU costs
- 31:16Closing
related talks
13mPreview: ML Training
08mHacks to Make LLM Training Faster
36mAccelerating Machine Learning at Scale with
34mOptimizing AI Inference with ML Compilers & Hardware
14mOptimize your TensorFlow Lite models
01h 02mWorkshop: LLMs Fine Tuning and Inferencing Using ON...
34mSpeeding up Your ML Workload: Pytorch Compile and Distributed Training
11mOptimizing Model Inference with PyTorch 2.0