Accelerating Generative AI
summary
Native PyTorch pipelines compile and fuse real kernels to cut overhead and hit near memory bandwidth limits: baseline LLaMA‑7B goes from ~25 to 107 tokens/s with torch.compile with mode=reduce_overhead, then to ~157 tokens/s with int8 weight-only quantization, to ~203 tokens/s with INF4 kernels, and to ~240 tokens/s when adding speculative decoding. Segment Anything and SDXL see ~2× latency gains and big memory drops on A100. Implementation is ~760 lines of Python, only PyTorch plus SentencePiece, and the code runs and reproduces the reported speedups.
AI-written summary · about
chapters (15)
- 0:00Accelerating generative AI
- 1:04Segmentation with Segment Anything
- 6:03Image generation with SDXL
- 7:28Text generation with PyTorch
- 11:22Torch compile boosts inference
- 13:54Triton kernels in decode phase
- 14:57Model bandwidth utilization
- 17:03Weight-only quantization
- 19:08Speculative decoding explained
- 21:37AMD GPU optimizations
- 22:42INT4 quantization with torch export
- 23:48Tensor parallelism in PyTorch
- 24:31All optimizations combined
- 25:58Summary and takeaways
- 26:19Q&A
related talks
23mBlazing Fast GenAI Inference With Torch.compile
31mPyTorch 2.1 - New Developments
16mtorchtune: Easy and Accessible Finetuning in Native PyTorch
19mAI Kernel Generation: What's working, what's not, what's next
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
39mRunning On-Device AI With Qualcomm AI Engine Using LiteRT & ExecuTorch
29mKeynote
11mOptimizing Model Inference with PyTorch 2.0