Incident Management at Netflix Velocity
summary
Netflix reports that injecting failures with Chaos Monkey and a new Latency Monkey breaks assumptions: cranking latency to 500 ms across 100% of calls collapses a service during an in-progress migration and triggers a real outage. The company builds a centralized CORE SRE team, codified libraries for cache/persistence patterns, and a five‑minute regional evacuation. Lessons: chaos experiments reveal hidden retry/backoff amplification, blast‑radius mistakes, and that recovery planning beats exhaustive prevention.
AI-written summary · about
chapters (14)
- 0:00Introduction and context
- 1:51Netflix velocity and scale
- 5:33Winning moments of truth
- 7:06Evolution of chaos engineering
- 10:49Latency monkey experiment
- 15:02Lessons learned from failures
- 17:36New approach to failure
- 21:52Investing in resilience
- 25:55Expecting failure
- 31:01Incident management goals
- 34:28The core SRE team
- 38:31Incident response tactics
- 41:21Blameless post-mortems
- 42:21Q&A and concluding remarks
related talks
41mHow Netflix Embraces Failure to Improve Resilience and Maximize Availability
38mChaos Engineering: A Step Towards Resilience
30mAntics, drift, and chaos
49mChaos & Intuition Engineering at Netflix
36mChaos Engineering
25mIncident Analysis Before Chaos Engineering
23mHow to tame chaos effectively?
48mChaos Engineering: The Path to Reliability