Maintaining reliable systems
summary
Argues that outage impact shrinks by attacking three measurable windows: time to detect, time to repair, and time between failures, and notes human involvement typically adds 20–30 minutes to outages. Recommends aligning SLIs/SLOs and alerts to customer journeys, using automation and self‑healing, progressive canary rollouts with automatic rollback in CI/CD, Google’s IMOG incident command structure for clear roles, chaos engineering to find faults, and blameless postmortems that feed a shared repository to prioritize fixes.
AI-written summary · about
chapters (12)
- 0:00Reliable systems intro
- 1:04Welcome
- 2:05Incidents and impact
- 3:10Service Level Indicators and Objectives
- 4:12Reduce customer impact
- 5:35Detect time and alerts
- 6:39Repair time and response
- 8:04IMoG protocol
- 9:29Troubleshooting dashboards
- 10:11Failures and deployments
- 11:56Postmortems and culture
- 13:01Prioritization and resources
related talks
29mShrinking the Impact of Production Incidents
24mThe Formula for Faster Outage Recovery
30mImproving Reliability w/ Error Budgets & SRE
39mOops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash
22mDon’t wait for an outage to improve your reliability
51mShrinking Production Incidents
37mManaging incidents in microservices
42mYou put out the fire on production, what's next?