Evolution of Disaster Recovery @ Meta
summary
Meta documents a shift from a fixed extra-capacity DR buffer to a scalable site degradation strategy that deliberately sheds noncritical work to survive two-region outages, validated by production tests and auditing. Teams run zero‑notice single-region storms for years, begin full-scale Power Storms in 2022 that bring a region to absolute zero and recover it, and report encountering about 50 zombie services during one recovery. CI/CD dependency tracking with Bell jar and combined testing+auditing prove the approach works at scale.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 0:31Historical Context
- 2:21Single Region Failure
- 4:26Power Risks
- 11:27MultiRegional Outages
- 18:15Key Insights
related talks
30mDisaster Recovery Testing at Booking.com
29mBuilding Resilient Systems
57mChaos Engineering Primer
25mWe're Still Down: A Metastable Failure Tale
27mBuilding Ultra-Resilient Systems in Go
53mResiliency and Availability Design Patterns for the Cloud
46mKubernetes Stateful Application Disaster Recovery with Zero Data Loss
25mSmelling Smoke? How to Design a Disaster Recovery Strategy...