Managing Incidents
summary
Google-style incident process that assigns Incident Commander, Operations Lead and Communications/Planning roles, uses runbooks and BigQuery log queries, and treats about 70% change-induced outages as common. Response prioritizes short-term mitigations (rollbacks, capacity changes) to cut MTTR, with handoffs of five to thirty minutes and postmortems that produce tens to hundreds of blameless action items. Categorizes incidents as minor/major/huge and reserves huge for company-wide outages.
AI-written summary · about
chapters (11)
- 0:00Incident management at Google
- 1:29What incidents look like
- 4:18Defining incidents and escalation
- 6:24Organized response and categorization
- 9:56Unmanaged responses fail
- 11:44Google’s incident roles
- 16:38Planning lead and post-incident work
- 19:05Mitigation versus fixes
- 21:50Postmortems and blameless analysis
- 24:57Reliability versus features
- 25:59Summary
related talks
20mHow to Have an Operational Incident
28mHow to Have an Operational Incident (A Crash Course)
19mHow to handle incidents at scale
39mNavigating the Chaos: A Holistic Approach to Incident Management
32mIncident Management - Talk the Talk, Walk the Walk
01h 04mIncident Management - Talk the Talk, Walk the Walk
45mFailure Happens: Improving Incident Response in Large-Scale Organizations
30mHow To Have An Operational Incident (a crash course)