Lightning Talk: Achieving SOTA GEMM Performance: A CuTeDSL Backend for PyTorch Induc...
summary
Adds a CuTeDSL GEMM backend to PyTorch Inductor that queries NVIDIA’s Cutlass kernel library, prunes candidates with the NVMAL heuristic to about 5 candidate kernels, JIT-compiles and benchmarks them alongside Triton and cuBLAS, and selects the fastest for runtime. Supports standard/grouped GEMM plus MXFP8 and NVFP4 scaled MM, shows end-to-end results like ~7% speed up on Llama 3.370B for batch 16, and plans epilogue fusions and AOT.
AI-written summary · about
related talks
28mGenerating State-of-the-Art GEMMs for Heterogeneous Hardware with...
09mLightning Talk: Harnessing NVIDIA Tensor Cores: An Exploration of CUTLASS & OpenAI..
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
13mLightning Talk: Adding Backends for TorchInductor: Case Study with Intel GPU
16mLightning Talk: Efficient Inference at the Edge: Performance You Need at the Lowest...
08mA Halide Backend for TorchInductor
10mLightning Talk: Improved GEMM and SDPA Performance on ROCm With Composable Kernel
15mLightning Talk: AOTInductor: Ahead-of-Time Compilation for PT2 Exported Models