Help, My Datacenter is on Fire
summary
March 10, 2021 an OVH campus fire in Strasbourg knocks out a site that contributes to 3.6 million affected websites but the team contains impact by failing over most services to secondary sites, manually changing DNS for a stuck VIP and rebootstraping a nonredundant stack at another supplier from a two hour-old backup to be live by 09:00, recovering from 13 physical servers and 135 VMs lost. Key problems are a 60+ minute DNS update delay, hard‑coded IoT IPs and an MTU issue, solved by automation + infra‑as‑code and multi‑cloud, multi‑campus redundancy.
AI-written summary · about
chapters (7)
- 0:00Introduction and incident start
- 2:56Immediate impact assessment
- 5:23First technical problems and fixes
- 8:11Recovery and coordination efforts
- 13:07Long-term recovery and new DR plan
- 17:22Why automation saved us
- 23:02Lessons learned and conclusion
related talks
34mDatacenter Fires and Other 'Minor' Disasters
47mHow (not) to survive a data center outage
33mDatacenter Fires and Other "Minor" Disasters
40mThe World Blew Up but We’re All Okay: How We Managed a...
26mThe Big One
41mMonitoring OVH: 300k Servers, 27 DCs, a Metrics Platform
32mMonitoring OVH: 300K servers, 27 DCs one unified metrics platform
22mHoney, I Shrunk the Data Center!