How Netflix Embraces Failure to Improve Resilience and Maximize Availability
summary
Netflix builds a Simian Army that deliberately breaks production: Chaos Monkey hard‑kills machines (over 60,000 instances to date) to force stateless designs and autoscaling groups across AZs; Chaos Gorilla simulates AZ failures quarterly and reveals control‑plane, load‑balancer and directory self‑preservation bugs; Chaos Kong plans whole‑region drills. Latency Monkey injects delays and error rates while a Hystrix circuit breaker pattern and Cassandra clusters absorb failures. Janitor Monkey reclaims stale cloud resources; observability (Atlas) and devs‑as‑operators prove essential.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:43Netflix infrastructure overview
- 2:48Availability at Netflix
- 4:39Designing for failure
- 9:58Chaos monkey
- 12:16Lessons from chaos monkey
- 14:20Chaos gorilla
- 18:30Chaos kong
- 19:00Latency monkey
- 20:43Hystrix circuit breaker
- 22:42Resilience testing strategies
- 25:02Janitor monkey
- 27:02Simeon army culture
- 30:25Netflix OSS initiatives
- 32:26Q&A session
related talks
30mAntics, drift, and chaos
55mIncident Management at Netflix Velocity
46mDesigning a System for Chaos
37mUse Chaos Testing to Build Resilience in the Software
36mChaos Engineering
35mChaos Engineering – from destructive monkeys ...
38mChaos Engineering: A Step Towards Resilience
49mChaos & Intuition Engineering at Netflix