Serving Machine Learning Models at Scale Using KServing
Talkat KubeCon + CloudNativeCon North America 2021 · on Thursday, 2021-10-14· at 14:30· Room 502 AB + Online·
summary
KServe adds a Model Mesh component that packs many models into shared runtimes and routes to them using a distributed model mesh LRU cache, reducing per-model overhead and Kubernetes limits. The demo loads 15,000 models on one 8‑CPU, 64GB node (experiments up to 20,000) with most requests served at single‑digit millisecond latency. Runtimes supported today include NVIDIA Triton and MLServer, with a PyTorch adapter planned, and KServe v0.7 exposes a single InferenceService CRD for both single‑model and multi‑model modes.
AI-written summary · about
chapters (16)
- 0:00Start
- 1:17Problem: production grade inference
- 1:58Production grade inference: How hard could it be?
- 4:09KServe overview
- 5:10Kserve Architecture
- 9:23KServe Standardized Inference Protocol
- 10:47And more advanced Metrics
- 11:19Current Serve model deployment
- 11:38Scalability limitations • Compute Resource limitation
- 11:51Compute Resource limitations
- 12:26Maximum pod limitations
- 13:53ModelMesh for Multi-Model Serving
- 14:39ModelMesh Architecture
- 15:47ModelMesh Components
- 16:47ModelMesh Features
- 20:46Preliminary Road Map
related talks
31mServing Machine Learning Models at Scale Using KServe
36mWhat's New, ModelMesh? Model Serving at Scale
28mExploring ML Model Serving with KServe (with fun drawings)
15mAI Model Inference on K8s
39mIntroducing KFServing: Serverless Model Serving on Kubernetes
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
29mIntegrating High Performance Feature Stores with KServe Model Serving
34mKServe in Production: Scaling Generative Inference the Cloud Native Way