Resilience & Chaos Engineering in the Cloud
summary
Expedia runs microservices on AWS and Kubernetes and demonstrates controlled chaos with a three-stage experiment: blackhole ElastiCache, inject 1s latency (below a 1.5s threshold), then 2s latency which trips the circuit breaker and makes reviews disappear; experiment runs against two pods in a sandbox using Gremlin. They validate fallbacks with Testcontainers and toxiproxy, store data in Amazon Aurora and observe operator atrophy—runbooks and alerts exist but hands-on practice (game days) actually matters.
AI-written summary · about
chapters (15)
- 0:00Introduction to talk
- 1:19About Expedia Group
- 2:24Limitations of checklists
- 7:04Problems with runbooks
- 9:57Operator atrophy in systems
- 11:43Common resilience practices
- 17:27Service mesh resilience features
- 18:54Release safety mechanisms
- 20:23What is chaos engineering
- 25:19Anatomy of chaos experiment
- 27:25Real-world chaos demo
- 33:52Post-experiment analysis
- 34:33State-of-the-art resilience
- 37:02Continuous verification concept
- 38:48Q&A and closing
related talks
19mChaos Engineering at Scale
22mAll Clouds Need a Sprinkle of Chaos
25mAutomating Chaos Attacks at Expedia
57mChaos Engineering Primer
35mImprove resilience with automated chaos engineering
40mJourney into the World of Chaos Engineering
29mImprove Resilience with automated chaos engineering
33mBeyond the Monkeys Chaos Engineering on the Cloud