Testing Before You Scale & Making Friends While You Do It
summary
Builds a Node-based load generation platform to simulate one account creating up to 20,000 incidents/day (≈7.2M/year) and runs three production nuclear option tests during weekly failure Friday experiments. First two tests cause moderate customer-impact and reveal throttling, downstream backlog, buggy generator code and spin-up multiplication issues. Outcome: modular, team-maintained tool, cross-team playbooks for ramping and isolating services, and ongoing production chaos experiments that surface real bottlenecks before customers do.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:35Nuclear Option project
- 3:37Testing in production
- 5:23What is chaos engineering
- 6:07Testing system bottlenecks
- 7:31Unknown scaling challenges
- 9:38Designing Nuclear Option tests
- 11:02Scientific method in testing
- 13:27Formulating hypotheses
- 14:27Running the experiment
- 15:50Results and challenges
- 18:42Cross-team collaboration
- 19:47Key takeaways
- 21:52Q&A: Stakeholder buy-in
- 23:39Q&A: Staging limitations
- 25:03Q&A: Chaos at PagerDuty
related talks
56mTesting at scale
48mDevops@Scale
42mPlanning To Fail
48mHow to not Crash in Prod, Load Testing Crash Course
25mGame time: A playbook to (unsuccessfully) 10x in a week and (successfully) 10x in a year
06mPrevent, Rather than Fix: Scaling to Enable Customers
43mBest practices for continuous testing and observability
57mLearning to Love your Non-functionals