Optimizing Model Inference with PyTorch 2.0
summary
Cloud-focused walkthrough of PyTorch 2.0 techniques: int4 quantization cuts memory enough to run a 12B model on smaller GPUs with only ~0.01% accuracy loss, saving cost; torch.compile + CUDA graph capture tuned from “full” to “full and piecewise” for decoding yields about 15% lower tail latency at P95; a head-to-head of attention kernels finds the paper’s SAGE attention 20% faster than diffusers’ native version so the solution is to patch the older SAGE attention into the newer diffusers for both speed and features.
AI-written summary · about
chapters (9)
- 0:00PyTorch inference optimization intro
- 1:23Model optimization KPIs and SLAs
- 2:48Three layers of inference optimization
- 3:10Quantization techniques and trade-offs
- 4:53PyTorch 2.0 compilation strategies
- 6:40Custom tensor ops challenges
- 8:46Optimization choices explosion
- 9:27Modular optimization framework
- 10:30Business-specific optimization solutions
related talks
32mTaming the Machine: Basics of ML Models Training and Inference Optimization
16mLightning Talk: Efficient Inference at the Edge: Performance You Need at the Lowest...
27mAccelerating Generative AI
14mOptimize your TensorFlow Lite models
54mKeynote: PyTorch 2.1 Technical Deep Dive
20mFireworks AI: Go from Hype to High - Value AI
35mClient Side Deep Learning Optimization with PyTorch
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile