Brainstorming Failure
summary
Describes applying Failure Mode Effects Analysis to production monitoring at Grub Hub: assemble cross‑functional teams, map the process, brainstorm every failure mode, list effects, and score severity, occurrence and detection to compute a risk priority number. Prioritize fixes that reduce detection, likelihood or severity, prefer self‑healing controls, add concrete metrics and feedback loops so failures show up before customers do. Mentions MindNode, a Google Doc template, New Relic and DataDog.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 2:00About Jeff
- 2:40Metaphor
- 3:02Internals
- 3:20Cockpit
- 3:57Feedback
- 5:10Failure Mode Effects Analysis
- 6:20Crossfunctional Teams
- 7:15The Process
- 8:53Brainstorming
- 10:24Risk Factor Scale
- 11:58Severity Ranking
- 12:54Occurrence Ranking
- 13:34Detection Ranking
- 14:32Risk Priority Number
- 14:51RPM Number
- 15:11Action Plan
- 16:52Feedback Loop
- 17:54Leading and lagging indicators
- 18:44Tools
related talks
44mPlanning for & Handling Failures from Open Hardware, Aviation, to Production at Google
41m‘Failure’ as Success: The Mindset, the Methods, and the Landmines
51mEmbracing Failure and Learning from Our Mistakes with Effective Post Mortems
41mDetective Hat: Investigating Production Issues
23mHow to tame chaos effectively?
40mRisks and problems: a healthy approach
16mInterview
15mPragmatic Incident Response: Lessons Learned from Failures