Envoy Proxy: Evolved for Serving LLMs
summary
Envoy adds a bodybased router that parses JSON request bodies (appending an x-gateway-model-name header) and an endpoint picker extension that scrapes Prometheus metrics and ranks endpoints; it also ingests Orca load signals from model servers and reports to LRS. Benchmarks with 10 model servers on H100 GPUs serving Llama 2 7B show similar throughput but much lower per-output-token and tail latencies; integrations include Kubernetes Gateway API extensions, token rate limiting, and pluggable AI safety guardrails.
AI-written summary · about
chapters (12)
- 0:00Envoy Proxy for LLMs
- 1:05Customer needs for LLM serving
- 2:07LLM deployment patterns
- 4:12Envoy’s LLM data paths
- 4:33Challenges in LLM APIs
- 6:37Self-hosted load balancing issues
- 8:03Envoy Proxy enhancements
- 9:50Model-aware routing
- 14:06Inference load balancing
- 19:28Kubernetes gateway APIs
- 25:12AI safety and caching
- 25:53Future directions
