Building Inference Workflows with Tile Languages
summary
Claim: inference gets 15x lower cost per token on recent Blackwell hardware and a tile compiler maps NumPy-style array code to tensor cores. Tools actually used: a Python-facing CUDA tile language called Coutile tile compiler, a tile IR virtual ISA and NumPy-like APIs that load/store tiles and emit MMAs. Surprising numbers: beats cuBLAS/cuDNN on several inference kernels. Built artifact: open-source compiler and runtime coming to GitHub within weeks, working on FP4, debugging, and multi-architecture support.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:05Tile languages for inference
- 3:12Hardware evolution Hopper to Blackwell
- 6:28Why GPU programming changes now
- 10:16Grid vs thread-level GPU programming
- 13:05Missing block-level abstraction
- 15:36CUDA Tile Programming
- 17:49Tile model and Python integration
- 21:42Performance vs CUDA Python
- 25:12Key features of CUDA Tile
- 27:01Benchmarking vs state-of-the-art
- 28:25Q&A
related talks
01h 26mGPU Algorithm Authoring with CUDA Tile
01h 00mThe Shape of Things to Come
28mThe Future Is Tiled: Using CuTile & TileIR To Write Portable, High-performance GPU...
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
34mOptimizing AI Inference with ML Compilers & Hardware
34mPanel Discussion: Training and Inference at Planet Scale
14mWelder: Scheduling Deep Learning Memory Access via Tile-graph
29mUnlocking the Future of AI: Trends, Infrastructure, and Tokenomics