Database Incident Management: When Everything Goes Wrong
summary
Concrete incident rules and tooling decide what matters: prevent data corruption ranks above bringing systems back, backups run every 15 minutes and each restore is tested daily, on-call handoffs occur after six hours, alerting aggregates by shard not host, PagerDuty response plays and event‑intelligence reduce noise, a bash tool safely stops DB VMs, a script refreshes docs and creds for on‑call, use spreadsheets and CAN reports (conditions actions needs), binlog analysis but no CSVs on laptops, and push data fixes through peer‑reviewed code.
AI-written summary · about
chapters (5)
- 0:00Start
- 3:09Alerting Reduce the noise
- 12:15Assembling
- 34:28Responding
- 38:38Learning After the dust settles
related talks
51mDatabase Disasters and How to Find Them
19mHow to handle incidents at scale
26mMission Critical Mindset Building a Culture of Resilience in DevOps Teams
01h 04mIncident Management - Talk the Talk, Walk the Walk
28mHow to Have an Operational Incident (A Crash Course)
20mHow to Have an Operational Incident
37mIncidents are a new normal
22mAfter the dust has settled: learning from incidents