It's Down! Simulating Incidents in Production
summary
Built a Faraday middleware called response_modifier and a run_gameday feature flag, toggled by rake tasks to add a dozen users, then flip the flag to simulate a service returning 500. The injection causes the Rails app to crash for flagged users, Bugsnag surfaces errors but PagerDuty does not page; teams spend 40 minutes prepping expectations, then debrief to write runbooks, add Datadog dashboards and tags, and fix alerting and visibility gaps discovered during the exercise.
AI-written summary · about
chapters (10)
- 0:00Start
- 2:54Styling Engineering Stitch Fix
- 8:32CHAOS ENGINEERING AT STITCH FIX A CASE STUDY
- 9:21DEFINE SIMULATIONS
- 11:06Scenario: Service returns a 500
- 15:29GATHER EXPECTATIONS
- 16:49TALK THROUGH EXPECTATIONS
- 21:23REVISIT EXPECTATIONS
- 22:56WRITE & EDIT RUNBOOKS
- 23:49UPDATE DASHBOARDS
related talks
30mIt's Down! Simulating Incidents in Production
35mSite Reliability on Rails
36mContinuous Deployment with Rails
35mThe Virtuous Cycle: Getting Good Things out of Bad Failures
26mThe Sounds of Silence: Lessons from an 18 hour API outage
23mSquash Production Defects Quickly - The Power of Structured Logging in Rails
34mGuide to Continuous Deployment with Rails
22mBe prepared for production incidents