Learning from Firefighters to Improve Systems Reliability
summary
Adapts the Ten Standard Firefighting Orders (codified since 1957, pocket 120 page playbook) to SRE practices and shows concrete practices that reduce operator harm and outage impact. Uses chaos testing tools from the CNCF landscape (mentions 3–5 viable tools), game days, honeypots, and controlled burns, and recommends playbooks, documented orders and timestamps via incident tooling like Etsy’s morg and Backstage wizards. Gives hard examples: Man Gulch escalation from 60 to 3,000 then 5,000 acres and an EBS deletion -> 5 days recovery costed millions so quantify outages in Tableau or Looker.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:06Speaker background
- 3:59Man Gulch fire lessons
- 7:31Firefighting orders overview
- 8:54Keep informed on system
- 9:58Consider weather factors
- 11:24Chaos testing retrospectives
- 13:10Graceful degradation design
- 14:35Communication in incidents
- 15:40Proactive honeypots
- 16:22Controlled burns fatigue
- 17:49Documentation training
- 19:35Declare incidents early
- 21:18Clear written orders
- 23:31Root cause analysis plural
- 25:36Protect system aggressively
- 28:51Command and control
- 31:04Reliability depends on team
- 34:04Q&A
related talks
15mSustainable Incident Management for happy SRE teams
31mThe Art of Turning Incidents into Opportunities
25mUnleashing the potential of SRE for modern operations
25mSRE Guiding Principles & Transformational Framework
38mThe Smallest Possible SRE Team
25mExpect the Unexpected: Preparing SRE Teams for Responding to Novel Failures
22mLooking Back to Look Forward
25mBuilding a highly scalable system to assure Business Resilience