Serving Machine Learning Models at Scale Using KServe
summary
Explains how KServe replaces one-model-one-service with a Model Mesh multi-model serving design that co-locates many models in a single runtime, integrating Triton and MLServer and loading models from S3. Gives concrete overheads (example sidecar 0.5 CPU / 0.5 GB), compares ~40 models on a single 8-core node for one-per-service vs 20k models with model mesh, and reports single-digit millisecond latency sustained up to 1,000 QPS per node.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:14Background about KServe
- 1:40Milestones
- 3:00Model Development
- 3:51Challenges
- 6:10KServe
- 7:41KServe Components
- 9:33Standard Inference Protocol
- 10:08HTTP Protocol
- 10:37GRPC Protocol
- 11:00New Scalability Problem
- 11:17Current Approach
- 12:13Problem
- 12:51Compute resource limitations
- 14:05Maximum pod limitations
- 14:49Maximum IP address limitations
- 16:08Model Mesh Solution
- 19:06Performance Test
- 20:22Latency Test
- 21:26Model Mesh
- 21:44Roadmap
- 23:00Questions
- 29:42Original Design
related talks
40mServing Machine Learning Models at Scale Using KServing
36mWhat's New, ModelMesh? Model Serving at Scale
28mExploring ML Model Serving with KServe (with fun drawings)
29mIntegrating High Performance Feature Stores with KServe Model Serving
29mServing the Future: KServe’s Next Chapter Hosting LLMs & GenAI Models (with Fun Drawings!)
15mAI Model Inference on K8s
34mServing ML Models at Scale with Seldon and Kubeflow
32mEnhancing the Performance Testing Process for gRPC Model Inferencing at Scale