Building a best-effort distributed fallback
summary
Built a best‑effort distributed fallback that triggers in the gateway when upstream returns HTTP 500+, uses Envoy sidecar for fault injection and circuit breaking, and serves a reduced offer set from locally pulled geospatial config so users can still request rides during outages. Feature toggles live in a feature-toggle kit and can disable the fallback in ~2 minutes; continuous tests and a chaos experimentation UI exercise the fallback (staged percent runs) and acceptance tests confirm it works—trades degraded UX for sustained availability while avoiding DB overload via static stability and overprovisioning.
AI-written summary · about
chapters (17)
- 0:00Start
- 3:09The Purchase Flow domain
- 7:40Build a fallback?
- 9:33Problems with a distributed fallback
- 12:52Building Resiliency
- 13:14Improve reliability of the normal mode
- 13:59Sidebar: Fault injection testing
- 15:13Sidebar: Envoy service mesh
- 18:49Pick the simplest fallback design
- 21:42Sidebar: Static Stability
- 23:29Reuse code from the normal mode
- 24:36Continuously exercise the fallback
- 25:41Sidebar: Feature Toggles
- 28:48Thoroughly test the fallback-end to end
- 29:49Support disabling the fallback
- 30:40The fallback mode
- 32:06To recap...
related talks
42mDesigning for failure
15mBoosting System Resilience: Exponential Backoff
43mFault tolerance – look, it’s possible!
20mUber's Failover Architecture: Reconciling Reliability and Efficiency in Hyperscale...
15mFail Open, Fail Fast: Improving Envoy Resilience in Latency-Critical Systems
32mHow does your application behaves when everything goes wrong?
19mZero-Downtime by Design: High-Availability Migration Patterns
51mDesigning for Failure: Build Resilient Services