The Big One
summary
Datadog suffers a global outage when an unattended systemd security update on Ubuntu restarts thousands of Kubernetes nodes and deletes their psyllium CNI tables, knocking every region offline simultaneously at 06:00 UTC. Recovery uses triage and 500+ engineers rotating over ~48 hours, with mitigations restoring EU1 by ~08:30 UTC, full infrastructure by ~19:00 UTC and data backfill over the next day; differences in cloud behavior (AWS nodes auto-deleted vs GCP/Azure not) and API rate limits slow recovery. Internal work streams product gets used in production and meaningfully helps coordination; status pages and product-specific communications see concrete redesigns.
AI-written summary · about
chapters (6)
- 0:00Introduction and context
- 3:20Incident trigger and initial response
- 7:09Incident response process
- 11:22Recovery challenges and differences
- 14:10Lessons learned and improvements
- 21:26Q&A session
related talks
40mThe World Blew Up but We’re All Okay: How We Managed a...
25mManaging expectations
34mDatacenter Fires and Other 'Minor' Disasters
41mKubernetes the Very Hard Way
23mDiving through RCAs a quest for reliability
27mWhen One Line Took Thousands of Websites Offline
20m[Almost] Fail & Tell: Stop The World
34m5 Things I Learnt from Causing a Cloud Provider Outage