Datacenter Fires and Other 'Minor' Disasters
summary
Midnight release pushes master instead of release branch and runs unapproved migrations, forcing a drop and restore from a full database backup and causing a three‑times‑normal outage; a collocated credit‑card processor goes offline after a data center fire over a holiday, but a checkbox in admin UI isolates payments so the site keeps working; planned power maintenance fries PSU hardware and systems ran on diesel for two weeks; malformed websocket messages (clients send 30–60 frames/s) cause a self DDOS until malformed input and exponential backoff are fixed.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:36Friday Hug
- 2:54If nothing bad happens you don't have good story to tell.
- 3:19War Stories
- 4:04Midnight
- 7:57Automate Everything
- 8:26Always Have A Backup
- 12:26Dependencies
- 13:15Fire #2
- 16:15No Silos
- 16:55Disaster Recovery
- 17:21Cloud
- 18:04Japan
- 19:57Assumptions
- 20:17Solder
- 23:53Enemy
- 24:27We all mess up
- 25:04What Saves Us?
- 25:55Learn
- 26:49Communication
- 27:40Ownership
- 28:06Postmortem
- 28:19Blameless
- 29:07Diversity
- 30:20Questions - Cat Pictures
related talks
33mDatacenter Fires and Other "Minor" Disasters
40mThe World Blew Up but We’re All Okay: How We Managed a...
26mA Tale of Two Outages
25mHelp, My Datacenter is on Fire
22mWhat can possibly go wrong? The story of an outrage at Booking.com
26mThe Big One
25m"The website's down!" Stories and lessons on keeping your website up.
30mWhat Breaks Our Systems: A Taxonomy of Black Swans