Lightning Talk: Slash LLM Cold-Start Times by Pre-distributing GPU...
summary
Pre-distribute compiled GPU kernel caches as OCI images so pods skip repeated JIT compilation and start minutes faster. A utility called Model Cache Vault (MCV) packages the local PyTorch GPU kernel cache into a signed OCI image, and a Kubernetes operator GPU Kernel Manager (GKM) pulls, verifies with cosign, extracts into PVCs and mounts them into inference pods. Requires same GPU architecture and inputs, needs one initial JIT run, and works with KServe for cluster-wide rollout.
AI-written summary · about
related talks
25mFrom Cold Start to Warp Speed: Triton Kernel Caching with OCI Container images
09m⚡ Lightning Talk: Advanced GPU-Orchestrated Workflows and HPC Integrations on K8s for Distributed AI/ML at Scale
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform
11mLightning Talk: Roche's Kubeflow Story: Large Models in Seconds, Costs Cut, Securit...
09mLightning Talk: Bringing Multi-Tenancy to GPU-Based Workloads
34mBeyond Static Pods: Dynamic GPU Sharing and Low-Latency Model Switching for LLM ...