Keynote: LLM-Aware Load Balancing in Kubernetes: A New Era of Efficiency
Talkat KubeCon + CloudNativeCon Europe 2025 · on Friday, 2025-04-04· at 09:06· Level 0 | ICC Auditorium·
summary
New Kubernetes APIs called Gateway API inference extension turn Gateways into inference routers and plug into Envoy via Envoy xDS callout to run a realtime scheduler that models per-request cost, GPU memory and token generation. LoRA adapters load alongside base models to enable adapter sharing that yields 1.5–4.7x GPU cost savings in Bytedance cases, and the load balancer steers by unused GPU memory to get 30% higher QPS at constant latency across heterogeneous clusters.
AI-written summary · about
chapters (10)
- 0:00LLM-aware load balancing intro
- 1:45LLM inference routing needs
- 3:31LLM serving challenges
- 5:38LoRA adapters for density
- 7:47Text-to-SQL with LoRA
- 9:15Balancing latency and cost in LLM serving
- 10:41Round-robin fails for LLMs
- 12:49Standard for LLM-aware infra
- 14:15Gateway API for inference
- 15:18Future of LLM load balancing
related talks
10mLightning Talk: Kubernetes Gateway API for Complex Environments and Service Providers
18mKeynote: Adapting Kubernetes for Machine Learning Workflows
35mIntelligent Load Balancing in Kubernetes
13mBuilding Reliable Infrastructure for LLM Workloads on K8s
20mInferencing LLMs in production with Kubernetes and KubeFlow
16mKeynote: An Open-Source Stack for AI Compute: Kubernetes + Ray + PyTorch + vLLM
31mScaling LLMs at Bloomberg Using Envoy AI Gateway: A Hybrid Cloud Approach
11mLightning Talk: Intelligent Traffic Routing for Distributed LLM Inference: Beyond Trad...