KServe in Production: Scaling Generative Inference the Cloud Native Way
summary
KServe presents a Kubernetes-native inference platform with concrete production features: distributed inference via tensor+pipeline parallelism (example: split a large model into eight GPUs across two nodes), KV cache offloading across GPU→RAM→local→remote, and a model cache that cuts model pull startup from ~10–15 minutes to near-instant. It supports KNative serverless, raw Deployment, and an LLMInferenceService with token-aware WBA autoscaler, EPP endpoint picker for KV-aware routing, OpenAI-compatible endpoints (example Qwen-2.5 7B), multi-tenant LoRA adapters, and container snapshot restore under ~10s.
AI-written summary · about
chapters (16)
- 0:00KServe in production intro
- 1:04KServe architecture core
- 3:33GenAI workload differences
- 6:00Multi-GPU scaling GenAI
- 8:29Traffic routing hybrid
- 10:34Deploy inference services
- 12:20Auto-scaling GenAI
- 13:02Local model cache
- 14:03Multi-node GPU setup
- 14:45KV cache offloading
- 15:50Gateway hybrid inference
- 17:14LLM inference GenAI
- 18:37Multi-tenant LoRA serving
- 20:01Roadmap unified fabric
- 22:52Community resources
- 23:35Q&A scaling cache
related talks
24mKServe Next: Advancing Generative AI Model Serving
15mAI Model Inference on K8s
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
28mExploring ML Model Serving with KServe (with fun drawings)
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
25mCloud Native GenAI Using KServe and OPEA
35mServing Large Language Models using KServe