Gotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
summary
Bloomberg implements a GPU-local model cache using KServe model-cache CRD and a node agent to auto-redownload missing models, plus a Karmada propagation policy and a model-caching API to federate caches across clusters. A 7TB LLM that takes ~15 minutes to boot from remote storage starts in ~1 minute from local cache and scaled pods warm GPU memory in ~45 seconds versus >10 minutes without caching. Persistent volumes back caches, autoscaling uses token-throughput metrics, and GPU hours drop significantly.
AI-written summary · about
chapters (13)
- 0:00Model caching for AI workloads
- 1:03AI at Bloomberg platform
- 3:52Model loading bottleneck
- 4:55KServe inference startup issues
- 7:07Model cache design
- 13:55Node agent for cache resilience
- 14:15Model caching benefits for LLM fine-tuning
- 15:59Karmada multicluster orchestration
- 19:51Propagating caches with Karmada
- 20:32Model caching API
- 21:35Extending cache to cloud
- 23:01Live demo of caching
- 28:06Practical benefits and future
related talks
21mOptimizing Model Serving on Kubernetes With Model Streaming
38mExploring Distributed Caching for Faster GPU Training with NVMe, GDS, and RDMA
14mBlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching
43mProblems and Solutions with AI Model Deployments in k8s
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
07mLightning Talk: Slash LLM Cold-Start Times by Pre-distributing GPU...
19mBeyond the Node: Scaling Inference with Cluster-Wide KVCache...
28mMaximize GPU Utilization for Model Training