The Overnight Failure
summary
A recounting of a real production incident where a billing system charged users thousands of times too much; it explains the root cause, immediate impact, and how the team responded. The talk covers the operational, management, and recovery steps taken to fix the outage and mitigate customer harm, including consequences for personnel and the company’s survival.
AI-written summary · about
chapters (9)
- 0:00Welcome
- 3:43Why failures matter
- 9:21Worst mistake ever
- 12:33Black Saturday timeline
- 17:35Technical causes
- 19:40User impact
- 21:46Lessons learned
- 24:13Handling crises
- 26:23Recovery and wrap-up
related talks
31mPixels, Promises, and Panic - Horror Stories of Production Nightmares
06m“The Power of No”
09mWhen I broke production, and it almost broke me
28mElephant in the Blameless War Room—Accountability
19mDev/Ops relationships, status: It's Complicated
36mGetting back to sleep as soon as possible for the on-call developer
41m12 Factors of Pain and Suffering
05mWho's to Blame?