Shrinking Production Incidents

on YouTube ↗51m

" summary

Defines failures with client-facing SLIs/SLOs and uses error budgets to decide tradeoffs, then forces fast detection via client or prober metrics, dashboards and prefabricated dashboard links. Shrinks repair time with tested runbooks, routine drills like wheel of misfortune, training and a pager-load rule of two incidents per shift (recommendation: six engineers per rotation). Prevents recurrence with CI/CD, gradual rollouts, automated rollbacks, chaos/disaster drills; moving from 99% to 99.9% reliability can cost ~10x more.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 1:29What is SRS
3. 6:34Outages
4. 11:49Postmortem
5. 16:53Blamelessness
6. 23:00Focus
7. 25:00Time to Detect
8. 26:12Refining SLA
9. 31:15Mitigation
10. 32:45Improving Mitigation
11. 38:15Responder Fatigue
12. 41:40Time Between Failures

" related talks

• 29mShrinking the Impact of Production IncidentsYuri Grinshteyn · Reliability @Scale Summer 2022
• 47mCultivating Production ExcellenceLiz Fong Jones · NDC DevOps 2020
• 29mReducing Trauma in ProductionMandi Walls, Julie Gunderson · Conf42 Site Reliability Engineering 2021
• 25mHow to detangle and understand productionLiz Fong Jones · The Lead Developer New York 2019
• 58mCultivating Production ExcellenceLiz Fong Jones · linux.conf.au 2022
• 15mMaintaining reliable systemsAyelet Sachto · Conf42 Chaos Engineering 2022
• 29mCultivating Production ExcellenceLiz Fong Jones · ITNEXT Summit 2019
• 27mGetting Started with SREStephen Thorne · DevOps Enterprise Summit London 2018