Panel: Routing Intelligence Vs Traffic Control: Arc...
summary
Gateway Inference Extension (GAIE) debates whether routing logic lives out-of-band as an external processing call or in-path as a next-hop Service. Practical constraints: requests can be large (32 KB typical, occasional 100 MB), Envoy today allows only one outbound call so you cannot do multi-step deferred decode, and sidecars are used to orchestrate prefill/decode. Proposed fixes include a coordinator/orchestrator server, agent-gateway rewrites, or changing xrock; KV cache tiering and HA fail-open behavior are tested tradeoffs.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 1:47LLM inference load balancing
- 5:20LLM routing architectures
- 6:03Massive LLM payload processing
- 8:14Modern routing proxies
- 9:41Unifying cloud inference APIs
- 10:43Semantic routing vs model selection
- 13:12Two-phase and three-phase processing
- 15:44Envoy disaggregation gaps
- 16:46LLM routing protocols beyond XRock
- 17:28Request orchestrators as gateways
- 18:33Monolithic vs microservices proxies
- 20:42Composable workload architectures
- 22:07Customer reluctance to topology changes
- 23:18Inference gateway deployment
- 25:08Inference gateway fallbacks
- 26:53Extension-based operational complexity
- 28:20External service ownership
- 30:26Deferred decode challenges
- 31:55Prefiller-first request routing
- 32:59Microservice disaggregation
- 34:26Creative solutions
- 35:58Gateway API compatibility
- 37:21Distributed KV cache impact
- 40:10Summary
related talks
31mBehind the Scenes of Building a Gateway API Dataplane...
27mEnvoy Gateway: An in-Depth Guide of Its Benefits, Use Ca...
36mGateway API The new way to travel north south
27mPanel: Tales from the Edge: Is the Edge More Important Than the Service Mesh?
20mEnvoy Gateway Project Update
29mThe rise of Layer 7, microservices, and the proxy war...
32mOne API To Rule Them All? What the Gateway API Means For Service Meshes
37mJourney Through Kubernetes Ingress: Past, Present, and Beyond