We're Still Down: A Metastable Failure Tale
summary
A globally distributed CDN enters a metastable failure when a DR test plus cross‑region transcoder compression causes CPU spikes, OOMs and health checks to fail, producing thundering herd oscillations between two copies of media across regions and hours of outage. The unorthodox fix is disabled health checks to force even load spread, which actually stabilizes recovery; followups include removing transcoder compression, using Envoy panic threshold/AWS target‑group configs, cgroups v2 limits and autoscaling, and the incident resolves.
AI-written summary · about
chapters (7)
- 0:00Understanding metastable failure
- 2:21Incident onset and observation
- 4:37CDN architecture overview
- 9:30Recovery via health check removal
- 14:39Root cause analysis
- 18:01System design improvements
- 24:07Key takeaways and conclusion
related talks
37mWhen the Cure Is Worse than the Disease: Metastability in Recovery
38mIt Is OK to Be Metastable
22mEvolution of Disaster Recovery @ Meta
34mDistributed Systems - What Can Go Wrong and How to Get It Right
29mBuilding Resilient Systems
27mBuilding Ultra-Resilient Systems in Go
24mAverting crisis at multi-petabyte scale
05mPreventing cascading failures in a global network