Serving Large Language Models using KServe
summary
A demo of KServe on Kubernetes (OpenShift on AWS) showing orchestration of a model server that pulls a Hugging Face Granite model via an init container and runs in a pod with a BLLM model server plus Keetip/Istio sidecars. The presenter reports a 2B model runs locally from a Docker Hub container image, an 8B model fits but uses about 14 GB of GPU memory leaving no headroom, and KServe exposes OpenAI‑style APIs, supports canary releases and inference‑graph wiring but does not perform fine‑tuning.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:12AI model lifecycle
- 6:51Operating AI models
- 14:02Predictive vs generative
- 18:17Local LLM demo
- 23:27Deploy models on KServe
- 30:18KServe resource needs
- 32:09KServe vs cloud services
- 33:55Q&A
related talks
31mOvercoming Challenges in Serving Large Language Model
28mExploring ML Model Serving with KServe (with fun drawings)
34mImprove AI Inference (serving models) With KServe and VLLM
15mAI Model Inference on K8s
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
24mKServe Next: Advancing Generative AI Model Serving
14mAI Deployment: Mastering LLMs with KFServing in Kubernetes