Datacenter Fires and Other "Minor" Disasters
summary
Midnight release corrupts production after pushing master instead of the release branch and running migrations then a 45-minute restore from a backup recovers the site; a data‑center fire takes credit‑card processing offline for four days without credit cards so the system falls back to a switchable free‑play mode; colo maintenance triggers a halon evacuation and an 11‑day diesel run while racks get rewired; a WebSocket bug at 30–60 frames/sec causes a self‑inflicted DDoS; specifics, numbers, and fixes drive the decisions.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:30War Stories
- 1:36Midnight
- 6:04Automate Everything
- 6:33Always Have A Backup
- 10:16Dependencies
- 11:12Fire #2
- 16:01Disaster Recovery
- 16:25Cloud
- 18:49Assumptions
- 19:09Solder
- 23:54Enemy
- 24:18We all mess up
- 24:39What Saves Us?
- 25:01Learn
- 25:41Communication
- 25:55Ownership
- 26:18Postmortem
- 26:49Blameless
- 28:17Diversity
related talks
34mDatacenter Fires and Other 'Minor' Disasters
40mThe World Blew Up but We’re All Okay: How We Managed a...
25mHelp, My Datacenter is on Fire
25mThe day I broke production and it was on national television
10mGreat Caching Disasters!
21mHandling a Major Outage Transparently
29mBlack Friday - Lessons in Resiliency and Incident Response at Shopify
26mThe Big One