What Breaks Our Systems: A Taxonomy of Black Swans
summary
Conference postmortems group outages into six actionable failure modes and show concrete defenses. Examples include an ext3 2 TB filesystem ceiling that halts an RDS MySQL instance, PostgreSQL transaction-ID wrap requiring table truncation, an AWS per-cluster connection limit that cripples a mail provider, a Slack reconnection spike disconnecting 13% of users, a regex-driven CDN wipe at scale, and NotPetya hitting Maersk for about $300M — defenses: canaries, fast rollbacks, write-heavy load tests, circuit breakers, 3-retry exponential backoff, and tight dependency isolation.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:35Defining black swan events
- 2:00Defending against outages
- 3:35Overview of incident types
- 4:26Hitting system limits
- 7:48Defending against limits
- 9:37Spreading slowness
- 11:39Defenses for slowness
- 13:44Thundering herds
- 16:24Mitigating coordinated demand
- 19:40Automation interactions
- 21:31Controlling automation
- 23:02Cyberattacks
- 25:14Dependency problems
- 26:46Building resilient systems
- 28:01Incident management
- 29:14Conclusion
related talks
31mWhat Breaks Our Systems: A Taxonomy of Black Swans
29mBuilding Resilient Systems
16mEngineering Failure-Resilient Systems
34mDistributed Systems - What Can Go Wrong and How to Get It Right
29mHorrors of Distributed Systems
29mBlack Friday - Lessons in Resiliency and Incident Response at Shopify
01h 03mWhispers in the Chaos: Monitoring Weak Signals
28mReliability Lessons from Running High Traffic Services for More Than 10 Years