Planning for Failure using Chaos Engineering
summary
Claim: chaos engineering techniques can continuously assess production readiness by running controlled experiments that prove or disprove concrete hypotheses. Example experiment: set a timeout at 300ms for service A call to service B then inject 5s latency between service B and its database and observe whether A errors after ~300ms or waits; demo shows normal 0.6s, injected 6s, then enforced timeout yields 500. Tools: tc, stress-ng, chaos toolkit, AWS Fault Injection Simulator. Organizational items: tabletop wargaming and a war game champion coordinate runbooks and load generation.
AI-written summary · about
chapters (12)
- 0:00Planning for Failure using Chaos Engineering
- 4:23Systems Fail
- 8:39Example experiments
- 10:02A formal chaos engineering experiment
- 18:15Pre-requisites
- 18:42A "hypothetical" scenario
- 19:24People are systems too!
- 21:01Key points to remember
- 21:39Fundamental tools
- 25:59Chaos Toolkit
- 26:21AWS Fault Injection Simulator
- 28:01Key Resources
related talks
26mBreaking Systems for Resilience
18mChaos Engineering: Break It On Purpose
01h 02mPractical Chaos Engineering breaking things on purpose to make them more resilient against failure
38mChaos Engineering
47mImproving Business Resiliency with Chaos Engineering
48mChaos Engineering: The Path to Reliability
23mHow to tame chaos effectively?
57mChaos Engineering Primer