torch.compile and Diffusers: A Hands-On Guide to Peak Performance
summary
Shows how to use torch.compile with Diffusers to speed up diffusion models like Flux‑1‑Dev, including concrete configs: compile the Diffusion Transformer for ~1.5x inference speed on H100, use fullgraph=True to detect graph breaks, apply regional compilation to cut cold compile time ~7x (67s → 9.6s), mark tensors dynamic=True to avoid recompiles, and pair NF4 quantization plus CPU offload to drop memory from ~33GB to ~12.2GB; also explains LoRA hot‑swap patterns that avoid recompilation and reports a ~2.05x speed gain on Flux adapters.
AI-written summary · about
chapters (6)
- 0:00Introduction to diffusers
- 3:58Understanding torch.compile
- 8:45Optimizing graph performance
- 13:44Regional compilation
- 15:29Feature integration
- 20:04Advanced features and testing
related talks
23mBlazing Fast GenAI Inference With Torch.compile
10mLightning Talk: Diffusers: Bringing Cutting-Edge Diffusion Models to the Masses
01h 16mOptimizing workloads with Transformers and Diffusers
36mDL Compiler Panel Discussion
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
15mPyTorch 2.0: Unlocking the Power of Deep Learning with the Torch Compile API
09mTogether Goes Brrr: Threading Research & Production with Torch Compile
19mYou Might Not Need 50 Diffusion Steps