Three years of breaking things to make them better
summary
Three years of weekly fault injection in production called Failure Friday reboots random instances, suspends processes, and uses iptables to add 500ms latency + 25% packet loss to expose missing timeouts, DB locks, and boot-order fragility. Teams run planned experiments (one machine → one AZ → whole DC), keep incident commanders and scribes, log findings to JIRA, and built chaos cat to automate selected steps after years of manual practice; results are concrete fixes, runbook improvements, and better alerting.
AI-written summary · about
chapters (14)
- 0:00Start
- 0:26Conclusions
- 3:14What is Failure Friday?
- 4:09Our Basic Attacks
- 8:03What does it look like?
- 9:28Larger-scale events
- 11:16Benefits of Failure Friday
- 12:56How to get started
- 18:37Game Day Hour - Number 1
- 23:09Chaos Cat
- 25:42Team-specific FFs
- 29:04More Game Days
- 31:07Capacity testing
- 31:29Error budget
related talks
42mPlanning To Fail
01h 02mPractical Chaos Engineering breaking things on purpose to make them more resilient against failure
47mChaos Engineering Bottom Up
57mChaos Engineering Primer
26mFrom experimentation to continuous verification
39mOops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash
30mA Retrospective: Five Years Later, Was Chaos Engineering Worth It?
34mChaos Engineering:The good, the bad & the ugly