Optimizing Metrics Collection & Serving When Autoscaling LLM Workloads
Talkat KubeCon + CloudNativeCon Europe 2025 · on Thursday, 2025-04-03· at 17:30· Level 1 | Hall Entrance N10 | Room G·
summary
Demonstrates that switching to push-based metrics with KServe, an OpenTelemetry collector sidecar and a KEDA OTLP add-on yields much faster, configurable autoscaling for LLM inference by using GPU-native signals (KV cache percentage) instead of CPU/RPS; demo generates load with hey, scales replicas up from 1 to 4 within seconds but hits a two‑GPU limit, so node autoscaling via Cluster API (and GPU operator for drivers) is required; also uses short‑term metric storage and a 20‑minute stabilization window to avoid thrash.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:53Kubernetes autoscaling
- 5:29LLM autoscaling challenges
- 8:50Existing autoscaling limits
- 13:51OpenTelemetry and KEDA
- 17:43Demo setup and architecture
- 23:14Autoscaling with custom metrics
- 26:45Scaling with Cluster API
- 30:14Q&A
related talks
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
36mAutoscaling Isn't Set-and-Forget: Tuning KEDA & Karpenter for Real Workloads.
07mTidal Auto-scaling for Training and Inference Based on Kubernetes + KEDA
24mKServe Next: Advancing Generative AI Model Serving
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
31mShared GPU Scheduling & Proactive Autoscaling: A Production Blueprint for 1000+…
14mAI Deployment: Mastering LLMs with KFServing in Kubernetes
20mInferencing LLMs in production with Kubernetes and KubeFlow