AI Traffic Is Different: Rethinking Proxy Arc...
summary
Claims that traditional proxies fail for long-lived, GPU-backed inference and present a working Envoy-based gateway: control plane (Kubernetes + Gateway controllers), data plane (Envoy + external processor + rate limiter), and features like token-aware traffic control, dynamic model rollouts (2/10/50/100%), header-driven routing via XAI_EG_model header, provider request rewriting and failover, quota checks before dispatch, and observability exporting OpenTelemetry token metrics to Prometheus; demo shows these pieces integrated and operational.
AI-written summary · about
chapters (9)
- 0:00AI traffic and Envoy intro
- 1:05AI vs traditional API traffic
- 3:56Why proxies fail for AI
- 6:27Proxy features for AI traffic
- 9:16Envoy adapts to AI traffic
- 12:46Envoy AI gateway for LLMs
- 15:35Envoy AI gateway architecture
- 23:24AI traffic and proxy takeaways
- 24:51Q&A on Envoy AI gateway
related talks
15mTaming AI Sprawl: Your First Look at the Envoy AI Gateway
32mExpl(AI)n Like I’m 5: An Introduction To AI-Native Networking
17mThe Future of AI Traffic: What's New in Envoy AI Tra...
31mWhen LLMs Hit Production: Why You Need an AI Gateway
26mAI in My API Gateway ... But Why?
04mEnvoy Today: What’s New In Managing Cloud-Native And AI Workl...
37mEvolution of Envoy AI Gateway
27mEnvoy Proxy: Evolved for Serving LLMs