Evolving KServe: The Unified Model Inference Platform For Both...
summary
KServe announces a release that extends predictive-serving features to LLMs with a purpose-built llm-d CRD, support for disaggregated inference (separate model, tokenizer and KV-cache planes), and integrated KV-cache and prompt caching to cut latency and costs. It adds runtimes like vLLM, TensorRT-LLM and OpenMMLab, uses KEDA metric-based autoscaling (including scale-to/from-zero), and plugs into the Envoy AI Gateway. Project status: CNCF incubating, 19 maintainers and ~300 contributors.
AI-written summary · about
related talks
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
28mExploring ML Model Serving with KServe (with fun drawings)
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
40mServing Machine Learning Models at Scale Using KServing
26mStreamlining AI Pipelines With Elyra: From Development To Inference With KServe & VLLM
20mInferencing LLMs in production with Kubernetes and KubeFlow