KServe Next: Advancing Generative AI Model Serving
summary
KServe v0.17 introduces a dedicated LLMInferenceService CRD with model and KV caching, integration with the Envoy AI Gateway, and RMD integration for intelligent routing and disaggregated prefill/decode serving. Autoscaling can use GPU- and cache-aware metrics (not just concurrency), notably KV-cache utilization, and enforces token limits for multi-tenant rate control. Deployments support single-node models (suitable for <70B) or leader-worker multi-node parallelism; variant autoscaling and unified inference fabric are listed as near-term roadmap items.
AI-written summary · about
chapters (11)
- 0:00KServe Next intro
- 1:49KServe’s CNCF journey
- 3:15Generative AI serving
- 5:02Large model challenges
- 7:12KServe architecture
- 9:20LLM autoscaling
- 11:51Traffic management
- 13:35RMD for LLM
- 17:37LLM workload config
- 20:27KServe roadmap
- 22:56KServe community
related talks
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
34mImprove AI Inference (serving models) With KServe and VLLM
28mExploring ML Model Serving with KServe (with fun drawings)
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
31mThe Explorer's Guide To Cloud Native GenAI Platform Engineering
35mServing Large Language Models using KServe