Integrating Stateful Load Balancing At Scale In Envoy
summary
Databricks integrates Dicer autosharder with Envoy to route keys to in-pod caches, achieving 90-95% cache hit rates and supporting 99.95% availability while preserving horizontal pod autoscaling. Initial design uses ExtProc + sidecar (Scala) where Envoy queries a local process; that reduces DB load 10x for Unity Catalog but causes 5x throughput loss and P99s of 10–30 ms. Moving to Envoy dynamic modules yields ~10x lower tail latency, lower CPU, better fallback control, and upstream contributions to ExtProc and the Rust SDK.
AI-written summary · about
chapters (7)
- 0:00Databricks and stateful load balancing
- 4:56Stateful services and in-pod caching
- 8:52Envoy and Dicer integration
- 13:04Scaling challenges and performance
- 17:20Dynamic modules for performance
- 20:50Key takeaways and future
- 21:52Q&A on modules and adoption
related talks
26mRunning Envoy at the Edge
10mLightning Talk: Envoy @ Netflix
38mEnvoy Deep Dive
35mIntelligent Load Balancing in Kubernetes
33mGet to know Envoy - the modern proxy for cloud-native infrastructure - Baptiste Collard
34mIntro: Envoy
15mFail Open, Fail Fast: Improving Envoy Resilience in Latency-Critical Systems
33mRunning Envoy as an Edge Proxy