Embracing Collaborative Chaos
summary
Platform runs hundreds of microservices owned by 60 delivery teams across seven platform teams and experiments with planned disruptions called Chaos Days inspired by Chaos Monkey and AWS GameDay. A Terraform change once removes production databases across AZs and takes three days to recover by manually porting staging data. Typically five chaos days reveal gaps in logging, alerts and runbooks; first run required two to three days to restore. Start with 3–7 scoped experiments in staging with alerts enabled and timebox failures.
AI-written summary · about
chapters (23)
- 0:00Start
- 3:55Delivering vital, high traffic services, fast
- 5:03Building planet-scale, services, fast
- 5:32Operating on the edge of chaos
- 8:23How can your system fail?
- 9:24Addressing the risk of failure
- 11:43resilience
- 13:09Manual chaos
- 14:36Automated chaos
- 18:14Four chaos engineering approaches
- 20:44How does it help?
- 28:36Agents of chaos
- 30:25Chaos scope - know thyself
- 31:11Chaos scope - trust the brains-storm
- 31:54Chaos scope - brainstorm, then plan the detail
- 33:25Chaos scope - hack the chaos
- 34:17Deciding where
- 36:22Deciding when
- 37:20Keep calm and chaos on (agents)
- 38:35Keep calm and chaos on (everyone else)
- 39:11Divide and conquer, then regroup
- 40:56What did we learn?
- 42:37What's your next chaos step?
related talks
46mEmbracing Collaborative Chaos: Chaos Days on Large Platforms
01h 01mEmbrace chaos to achieve stability
47mChaos Engineering Bottom Up
28mAutomating Chaos Engineering GameDays with Terraform
03mChaos Day: When Reliability Reigns
45mDeveloping a Chaos Architecture Mindset
33mBeyond the Monkeys Chaos Engineering on the Cloud
01h 04mTrust and Confidence through Chaos