Cloud Native GenAI Using KServe and OPEA
summary
Shows a working pipeline that combines KServe and OPEA: Helm installs a doc‑summarization app plus a single VLM, Grafana dashboards report latency, token throughput and scheduler state, and KServe autoscaling uses HPA CPU thresholds to scale from 1 to 2 pods under load and scale down after ~5 minutes. Demonstrates model cache reuse to avoid re‑downloading terabyte models, and outlines an LLM InferenceService CRD for router/workload/KV‑cache layers that enable multi‑node inference, KV‑cache offload and hardware‑agnostic runtimes.
AI-written summary · about
chapters (10)
- 0:00Cloud-native GenAI intro
- 1:04LLMs hard to manage at scale
- 3:10LLM infrastructure challenges
- 7:53App and serving layers with KServe
- 11:49OPEA and KServe architecture
- 12:09Install OPEA with Helm
- 14:15Autoscaling OPEA with KServe
- 17:42Distributed inference features
- 21:57ON-AI unified API
- 24:43Summary and Q&A
related talks
31mThe Explorer's Guide To Cloud Native GenAI Platform Engineering
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
35mServing Large Language Models using KServe
24mKServe Next: Advancing Generative AI Model Serving
28mAI Pipelines With OPEA: Best Practices for Cloud Native ML Operati...
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
20mInferencing LLMs in production with Kubernetes and KubeFlow
33mFast Inference, Furious Scaling: Leveraging VLLM With KServe