Failing Over without Falling Over
summary
Uses STPA to enumerate failover hazards, links observability to control models, and treats near misses as data points. Recommends three-way replication across zones, automated game days and chaos engineering to surface issues, and tooling like CloudWatch, Prometheus and BigPanda for alert correlation. Advises operational rules: cut retries to zero at subsystem boundaries, telescope timeouts, build resilience before deploying apps. Reports early attempts fail often but repeated testing makes multi-zone and multi-region failover reliable.
AI-written summary · about
chapters (11)
- 0:00Introduction to failover challenges
- 1:07Availability theater and failure causes
- 2:53Building resilient systems with redundancy
- 5:26Control systems and failure models
- 8:57Control plane and disturbances
- 10:22Simplifying failover with symmetry
- 13:11Zone-level failover scenario
- 16:00Multi-region failover complexities
- 17:44Alert correlation and observability
- 19:09Preventing retry storms
- 20:10Continuous resilience testing
related talks
16mFrom System TAYE7 to High Availability: What, Why, How?
28mBuilding Resilient DevOps Pipelines: Strategies for Handling Unexpected Failures &...
30mFast, Available, Catastrophically Failing? Safely Avoiding Behavioral Incidents in Complex Production Systems
51mDesigning for Failure: Build Resilient Services
52mPerformance & Stability Anti patterns
36mDesigning for failure - architecting fault-tolerant system
55mGoing Resilient
10mBuilding an 'Oops-Less Operation' at Scale