Pragmatic Incident Response: Lessons Learned from Failures
summary
Advice is specific: run focused postmortems for small incidents and track the ratio of retros to declared incidents (ten minor incidents with two retros is a failure), set a goal to hold Sev1 retros the next day so memories remain fresh, stop paging on CPU or disk metrics and instead alert on symptoms not causes by using SLOs tied to user experience, rely on mean time to retro as a metric, use focused mitigation teams, and consider tooling like FireHydrant to automate pages and retros.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:05Small incidents prevent big failures
- 4:14Tracking mean time to retro
- 8:11Alerting on customer pain
- 11:01Who to notify
- 13:15Pragmatic incident response tips
related talks
25mPragmatic Incident Response: 5 lessons learned from failures
29mShrinking the Impact of Production Incidents
10mSimplify your postmortems and focus on scaling
22mAfter the dust has settled: learning from incidents
37mIncidents are a new normal
30mHow Did Things Go Right? Learning More from Incidents
15mSustainable Incident Management for happy SRE teams
15mIncident Ready: Chaos Engineer Your Incident Response