Developing a Chaos Architecture Mindset
summary
Argues cloud availability needs fast detection, regular failover tests and deliberate fault injection: DNS lapses, expired certificates and a flooded data center caused real outages; cross-region failover often takes days and failback months. Recommends game days for operators, automated experiments like chaos monkey deletes instances, tooling such as Chaos Toolkit for Kubernetes, Gremlin for traffic faults, Infection Monkey for lateral compromise and ChAP runs ~1,500 tests/day to catch regressions before customers notice.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 1:21What to do when something fails
- 2:06Permissions lookup fails
- 5:55Availability theater
- 7:58The fairy tale
- 8:58What goes wrong
- 11:40Renew DNS entries
- 13:18Flooded data center
- 14:37You cant legislate against failure
- 16:00Observability
- 16:19Network is reliable
- 17:27Book Chapter 2
- 18:40Chaos Architecture
- 19:24Interconnecting
- 21:09Applications
- 22:29Operators Users
- 24:18People Testing
- 24:54Who Runs the Fire
- 25:21Chaos Engineering Team
- 26:00Chaos Tools
- 32:00Security Red Team
- 33:51Security Tools
- 37:20Safety
- 38:40Safety anarchist
- 40:51Failures
- 41:32Safety Margins
- 43:01Hypothesis Testing
- 43:37WrapUp
