Lightning Talk: Intelligent Traffic Routing for Distributed LLM Inference: Beyond Trad...
summary
Kthena Router is a production-grade, Volcano-integrated router that replaces per-model gateways (10 models no longer means 10 gateways) and natively implements native PD disaggregation by routing prefill to prefill pools, tracking KV cache, and forwarding decode tokens to optimal decode nodes without sidecars. It uses a high-performance data plane plus control plane and a closed-loop control with adaptive modeling based on connections, token lengths, and load distribution. Pluggable policies include fairness, semantic-aware routing, KV-cache aware placement, and GPU utilization balancing, and it supports Gateway API and Gateway Inference Extension.
AI-written summary · about
chapters (10)
- 0:00Intelligent traffic routing intro
- 1:05Traditional gateway limits
- 2:10Cassini router for multimodal
- 4:17Kase workload orchestrator
- 4:37Per-model gateway overhead
- 6:04Prefill decode disaggregation
- 7:27Pluggable routing policies
- 8:10GPU utilization & KV cache
- 9:16K8s Gateway API integration
- 9:58Model canary with Kuma
related talks
14mSponsored Session: Lightning Talk: Efficient Inference Serving with Kubernetes Gateway...
26mYou Got a Match! LLM Prefix Aware Routing With Kubernetes
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…
16mKeynote: LLM-Aware Load Balancing in Kubernetes: A New Era of Efficiency
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
33mllm-d: Kubernetes Native Distributed Inferencing
29mMulti-Accelerator PyTorch Serving With NxD Inference and vLLM