Sponsored Session: Lightning Talk: Efficient Inference Serving with Kubernetes Gateway...
summary
Demonstrates a working stack: vLLM simulator plus the Kubernetes Gateway API Inference Extension and an agent gateway implementation that routes by header and body to pods hosting LLMs. Shows llama 3.1 and 3.2 replicas with model labels food_review_1 and food_review_2, an endpoint picker choosing pods that already loaded a model and are not busy, and a live demo that successfully routes requests; recommends one gateway per region to avoid cross‑region latency.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:05LLMs vs traditional servers
- 3:12Load balancer limits for LLMs
- 3:52Gateway API overview
- 4:56LLM inference extension
- 6:21Live demo setup
- 7:24Inference gateway config
- 8:27Header routing demo
- 9:54Body routing explained
- 10:57Q&A: Prefill and decoding
- 11:39Q&A: Multi-region setup
- 13:49Closing
related talks
16m⚡ Lightning Talk: Introducing LLM Instance Gateways for Efficient Inference Serving
10mLightning Talk: LLM-Driven Applications: Intro to LLM Agents in Kubernetes
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
11mLightning Talk: Intelligent Traffic Routing for Distributed LLM Inference: Beyond Trad...
10mLightning Talk: Make Way for the Gateway
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
32mScaling Generative AI Inference with llm-d
10mLightning Talk: Kubernetes Gateway API for Complex Environments and Service Providers