It's Down! Simulating Incidents in Production
summary
Code injects a fake 500 by adding custom Faraday middleware to override responses, toggled for a subset of users via feature flagging subset of users and flipped with a rake task toggles simulation. The game day runs in production, yields BugSnag and Datadog 500 spikes, exposes that the app actually falls over (unexpected full outage), and shows PagerDuty thresholds missed. Outcomes: linked runbooks to alerts, consolidated dashboards, backlog items for resiliency, and measurable proof that the middleware approach works.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 6:03Chaos Engineering
- 7:11Overview
- 8:45Simulating Services
- 12:52Feature Flags
- 16:08Prep
- 18:48Running the Command
- 20:48Simulation Results
- 23:10Key Learnings
- 24:31Process Resiliency
- 27:12Accessibility
- 28:39Mindset
related talks
27mIt's Down! Simulating Incidents in Production
22mDon’t wait for an outage to improve your reliability
22mBe prepared for production incidents
36mIt's not in Production Unless it's Monitored
23mNavigating in the Dark
35mThe Virtuous Cycle: Getting Good Things out of Bad Failures
42mPlan for Unplanned Work: Game Days with Chaos Engineering
34mMetrics driven development