⚡ Lightning Talk: Introducing LLM Instance Gateways for Efficient Inference Serving
Lightningat Cloud Native + Kubernetes AI Day Europe 2025 · on Tuesday, 2025-04-01· at 10:00· Level 1 | Hall Entrance N10 | Room G·
summary
A Gateway API extension called LLM Instance Gateways exposes model-aware routing by extracting the model name from request bodies and steering traffic via an endpoint picker service that scrapes live metrics (KV cache utilization, loaded adapters) from grouped model server pods in an inference pool. Specs plus reference implementations exist for Triton/VLM/TGI, support weighted rollouts and priority, and benchmark results are published in an upcoming Kubernetes blog post.
AI-written summary · about
chapters (10)
- 0:00LLM Instance Gateways intro
- 0:22Gateway API overview
- 1:45LLM vs web traffic
- 3:08What is inference and why it matters
- 4:55Use cases for inference
- 6:42Inference request flow
- 9:35Inference pool resources
- 11:42Model resource splitting
- 12:26Benchmarks and involvement
- 12:47Q&A on KV cache
related talks
14mSponsored Session: Lightning Talk: Efficient Inference Serving with Kubernetes Gateway...
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
26mYou Got a Match! LLM Prefix Aware Routing With Kubernetes
33mllm-d: Kubernetes Native Distributed Inferencing
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
10mLightning Talk: LLM-Driven Applications: Intro to LLM Agents in Kubernetes
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…