Antics, drift, and chaos

on YouTube ↗30m

" summary

Large distributed systems drift into failure despite correct code because local fixes, retries, logging, and support subsystems introduce new failure modes; the claim is that failure is inevitable but discoverable. Netflix uses Chaos Monkey and an automation platform ChAP to run controlled production experiments and automated canaries, routing small fractions of real traffic and watching metrics — e.g., they report six terminations a minute — and these experiments reliably reveal latent vulnerabilities before they cause outages.

AI-written summary · about

" chapters (43)

1. 0:00Antics, drift and chaos
2. 0:44Add a new test
3. 1:27Result: execution of unit test led to an outage
4. 1:41Moral: use unit tests sparingly, for they are dangerous
5. 1:53Complex systems exhibit unexpected behavior
6. 2:07System failure
7. 3:03Generalized Uncertainty Principle
8. 3:33Error handling
9. 5:40Latency increases
10. 6:43Support systems
11. 8:01Non-critical service failed
12. 8:26Lock shared by app threads
13. 8:41Lock contention
14. 9:09Memory leak bug in agent that monitors health of EBS servers
15. 9:27Mitigation
16. 9:45Command input entered incorrectly
17. 9:57Lorin's conjecture
18. 10:14Recap: Antics
19. 10:32Act II: Drift
20. 10:45Broken parts and sloppy devs
21. 11:26Drift into failure
22. 11:46Unruly technology
23. 11:56Software is hard to reason about
24. 13:54Scarcity and competition
25. 14:13Efficiency vs thoroughness
26. 14:58Decrementalism
27. 18:08Sensitive dependence on initial conditions
28. 19:27One day...
29. 21:09Recap: Drift
30. 22:17Make the wrong thing harder
31. 23:00Chaos engineering
32. 23:23Find vulnerabilities before they become outages
33. 23:50External validity
34. 24:08Risk: vulnerable to failure of non-critical services
35. 26:01Build a hypothesis around steady state behavior
36. 26:39Vary real-world events
37. 26:51Add latency to RPC calls
38. 27:07Route prod traffic to ChAP clusters
39. 27:29Integrate with deployment pipelines
40. 27:40Minimize blast radius
41. 27:59Takeaways
42. 28:122. Reasonable human decisions can lead to dangerous states
43. 28:27Chaos provides incentives

" related talks

• 38mChaos EngineeringNora Jones · LISA18
• 46mDesigning a System for ChaosPaul Stack · DevConf 2019
• 41mHow Netflix Embraces Failure to Improve Resilience and Maximize AvailabilityAriel Tseitlin · LISA '13
• 38mChaos Engineering: A Step Towards ResilienceNora Jones · REdeploy 2018
• 36mChaos EngineeringNikhil Barthwal · Agile India 2018
• 18mChaos Engineering: Break It On PurposeMorgan Carter · BSides London 2021
• 48mChaos Engineering: The Path to ReliabilityKolton Andrus · Agile India 2020
• 55mIncident Management at Netflix VelocityDave Hahn · LISA18