Introducing KFServing: Serverless Model Serving on Kubernetes
Talkat KubeCon + CloudNativeCon North America 2019 · on Tuesday, 2019-11-19· at 14:25· Room 15AB - San Diego Convention Center, Mezzanine Level·
summary
KFServing provides a Kubernetes CRD to serve TF PyTorch scikit-learn and custom containers, implementing CPU/GPU auto-scaling, scale to and from 0, and two-way canary rollouts; it plugs into Istio and KNative (uses the KPA with desired concurrency=1 for fast autoscaling). The project adds transformer and explainer components, integrates Alibi for model explanations, shows a Kafka-driven inference pipeline at Bloomberg, and is running in production while evolving the standardized HTTP/protobuf data plane.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:09Oh, you want to deploy your model on K8s?
- 2:49Introducing KFServing
- 5:11Putting the pieces together
- 5:42A Clean Interface
- 9:31Cloud Native Layers
- 10:31ML Service Meshes?
- 11:26Serverless Inferencing?
- 13:52Opinionated ML Microservices
- 15:53KF Serving on Bloomberg's Data Science Platform
- 17:33Serving Models on Production
- 19:26Bloomberg Production Requirements
- 21:20Platform Architecture
- 22:02Bloomberg Use Cases
- 23:32Inference Service with Transformer
- 25:19Custom Inference Service
- 25:57Inference Service with Alibi Explainer
- 26:40Inference Service with Text Explainer
- 27:39Inference on Kafka Streams
- 28:11Inference Router A/B Testing Proposal
- 29:22Production Model Deployment Process
- 31:09End to End Inference Demo
related talks
37mAccelerate and Autoscale Deep Learning Inference on GPUs with KFServing
10mBuilding Machine Learning Inference Through Knative Serverless...
30mAccelerate Federated Learning Model Deployment with KServe (KFServing)
28mExploring ML Model Serving with KServe (with fun drawings)
40mServing Machine Learning Models at Scale Using KServing
35mServing Large Language Models using KServe
14mAI Deployment: Mastering LLMs with KFServing in Kubernetes
42mDeploying ML at Scale with Kubernetes and TFX