From Cold Start to Warp Speed: Triton Kernel Caching with OCI Container images
summary
Packaging Triton-generated LLVM/CUBIN kernels as signed OCI images and pre-extracting them to nodes reduces JIT time and halves model cold-starts. A toolset: MCM for indexing, pruning and warm-generation, MCV for creating and cosign-verifying OCI cache images with hardware pre-flight checks, and GKM as a Kubernetes operator + CSI + daemonset that expands caches on nodes. Example run shows startup drop from ~115s to ~61.5s and torch compile time from 56.6s to 11.93s.
AI-written summary · about
chapters (8)
- 0:00Triton kernel caching intro
- 1:50Slow first run problem
- 2:57Solution with OCI images
- 3:42Cache Manager features & demo
- 8:33Model Cache Vault
- 11:18GPU Kernel Manager for K8s
- 17:41GKM performance gains demo
- 21:46Results summary & Q&A
related talks
07mLightning Talk: Slash LLM Cold-Start Times by Pre-distributing GPU...
35mFrom Minutes to Seconds: How we reduced ML models cold start time by 90% on Kubernetes
34mBridging the Gap: Compiling and Optimizing Triton Kernels Onto RISC-V Targets Based on MLIR
10mMaximizing Kernel Development Productivity Under Performance Constraints
07mLightning Talk: Production-Grade AI Isolation: K0s + Kata Containers for Zero-Tru...
25mLeveraging AI Ecosystem for Portable and Sustainable GPU Kernels in HPC
25mStreamlined Efficiency: Unshackling Kubernetes Image Volumes for Rapid AI Model...
32mEfficient Transparent Checkpointing of AI/ML Workloads in Kub...