Optimizing Model Serving on Kubernetes With Model Streaming
Talkat KubeCon + CloudNativeCon Europe 2025 · on Friday, 2025-04-04· at 11:00· Level 0 | ICC Capital Hall | Room J·
summary
Open-source Run:ai Model Streamer parallelizes reads and GPU transfers of SafeTensors so large weights stream into GPU memory instead of loading sequentially, cutting cold-starts for an Llama-3 8B (~15 GB) from minutes to seconds in benchmarks on an A10G GPU. Tests compare GP3, IO2 SSDs and S3; S3 experiments show sub-5s model load and GKE reports ~96% reduction versus direct download. Version 0.13 adds S3 native auth and GCS support; roadmap includes sharded and multi-GPU loading.
AI-written summary · about
chapters (12)
- 0:00Model serving optimization intro
- 2:37Cold start in AI inference
- 6:09Cost and latency of model loading
- 7:13Limitations of traditional loading
- 8:57RunAI Model Streamer intro
- 10:45RunAI Model Streamer features
- 12:13Benchmarking setup and metrics
- 13:59Benchmarking results
- 16:32Storage bandwidth impact
- 17:58Cloud storage caching
- 18:41Updates and roadmap
- 19:47Q&A
related talks
31mOvercoming Challenges in Serving Large Language Model
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
43mProblems and Solutions with AI Model Deployments in k8s
36mAccelerating High-Performance Machine Learning at Scale i...
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
31mFaster Model Serving with Ray and Anyscale
36mAccelerating Machine Learning at Scale with
38mAI in Kubernetes: How to Get Started?