Chaos Engineering: When the Network Breaks
summary
Demonstrations use Gremlin against a Kubernetes hipster-shop demo to run black hole and packet loss experiments with live outcomes recorded. Claim: chaos engineering yields measurable resilience and helped achieve a 10x reduction in incidents at scale. Results: black-holing the ad service gracefully removes ads and the site continues; black-holing the recommendation service causes a full outage due to a hard-coded dependency and single-pod fragility; injecting packet loss on the cart triggers 500 errors but retains cart data.
AI-written summary · about
chapters (12)
- 0:00Chaos Engineering intro
- 1:18Distributed systems complexity
- 3:06Creating chaos scenarios
- 5:11Selecting critical services
- 7:17Why mean time to detection matters
- 7:59Demo setup & testing
- 11:46Black hole on ad service
- 16:40Black hole on recommended products
- 20:38Packet loss on cart service
- 23:12Chaos experiments summary
- 24:15Questions after chaos
- 25:18Next steps for chaos
related talks
37mChaos Engineering: The Fine Art of Breaking Stuff in Production on Purpose
01h 02mPractical Chaos Engineering breaking things on purpose to make them more resilient against failure
38mChaos Engineering
39mChaos: Breaking your Systems to Make them Unbreakable
38mChaos Engineering: A Step Towards Resilience
46mDesigning a System for Chaos
18mChaos Engineering: Break It On Purpose
57mChaos Engineering Primer