Scientific and Safe Chaos Engineering
summary
Describes a production-first chaos program that starts with request-level failures using LinkedOut headers and feature flags before ever touching infrastructure, then moves to controlled node kills with Chaos Blade. Explains concrete policies: abort tests on SLA thresholds, double-check targeting to avoid opt‑ins of all users, and prefer shifting traffic between datacenters over full outages. Reports an empirical surprise: CPU and disk pressure both surfaced as increased network latency, so tests focus on client‑side failure handling first.
AI-written summary · about
chapters (13)
- 0:00Introduction to chaos engineering
- 1:02Resilience engineering foundations
- 3:10Resilience vs reliability engineering
- 8:34Chaos engineering validation
- 10:42Defining chaos engineering goals
- 15:02Establishing steady state
- 21:04Defining failure and success
- 22:52Lineage driven fault injection
- 25:45Control plane impact
- 28:55Safety third principle
- 31:44Strategic tech debt
- 33:12LinkedOut tool design
- 35:21Incremental chaos testing
related talks
38mChaos Engineering
23mHow to tame chaos effectively?
18mChaos Engineering: Break It On Purpose
57mChaos Engineering Primer
46mDesigning a System for Chaos
37mChaos Engineering: The Fine Art of Breaking Stuff in Production on Purpose
01h 02mPractical Chaos Engineering breaking things on purpose to make them more resilient against failure
48mChaos Engineering: The Path to Reliability