Fail Open, Fail Fast: Improving Envoy Resilience in Latency-Critical Systems
summary
Envoy sits in front of global object storage, implementing a decision matrix + policy engine that delegates auth and rate limits to an external outtrigger and toggles fail-open vs fail-close. Filter chains add ~2.68 ms of processing; co‑locating auth on LB nodes cuts end‑to‑end latency from 4–7 ms to ~0.65 ms. Load validation moves from K6 to Nighthawk for higher‑fidelity breaking‑point tests; changes roll out via shadowing and local caching to preserve 99.99% availability.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:44Envoy in front of storage
- 2:46Envoy processes signals
- 4:13Envoy at DigitalOcean
- 5:39Optimizing filter chain
- 7:27Collocating services
- 8:29Capacity planning
- 8:52Fail open vs fail close
- 10:41Kubernetes Envoy plans
- 11:47Nighthawk load testing
- 13:34Regional resilience steps
- 13:55Resilience results
related talks
25mBuilding Low Latency Topologies with Envoy
26mIntegrating Stateful Load Balancing At Scale In Envoy
34mIntro: Envoy
21mMaking Envoy Resilient to Sudden Increases in Load
10mLightning Talk: Envoy @ Netflix
27mEnvoy - Overview and Update: Senior Envoy Maintainer
33mGet to know Envoy - the modern proxy for cloud-native infrastructure - Baptiste Collard
14mKeeping Envoy Gateway On, Even When Bad Things Happen