Learning From Failure: Post Mortems
summary
Explains how to run a blameless post-mortem that turns a 17‑minute site outage (one example) and a kube cluster scale.sh mistake that causes ~1 hour total unavailability and >3 hours to fully resolve into concrete fixes: timelines, metric gaps, and prioritized tickets. Shows tools and mitigations actually used — timeouts on Facebook requests, dashboards, auto‑scaling, chaos‑testing, and the deployment system that did work — and argues for root causes (plural) analysis and raising an SLA to 99.5% SLA to generate meaningful followups.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:44SOMETIMES BUGS HAPPEN
- 1:04SOMETIMES OPERATIONAL FAILURES HAPPEN
- 2:08SOMETHING BROKE: WHAT DO?
- 2:40THE TEAN AND THE MEETINGS ARE FOR SOLVING PROBLENS, THERE ARE PLENTY OF OTHER VENUES WHERE PEOPLE DEVOTE THEIR CREATIVE ENERGIES TO SHIFTING BLAME WE DO NOT NEED ANOTHER ONE WE DONT SHOOT THE MESSENGER. ESPECIALLY WHEN THE MESSAGE IS. 1 SCREWED SOMETHING UP.
- 6:01WHAT'S A SYSTEMIC PROBLEM ANYWAYS?
- 6:47ROOT CAUSE CAUSES ANALYSIS
- 7:00HOW COMPLEX SYSTEMS FAIL: BEING A SHORT TREATISE ON THE NATURE OF FAILURE: HOW FAILURE IS EVALUATED: HOW FAILURE IS ATTRIBUTED TO PROXIMATE CAUSE: AND THE RESULTING NEW UNDERSTANDING OF PATIENT SAFETY
- 8:28WHAT DOES A POST-MORTEM LOOK LIKE?
- 20:36NATIONAL TRANSPORTATION SAFETY BOARD (NTSB)
related talks
51mEmbracing Failure and Learning from Our Mistakes with Effective Post Mortems
28mAfter the dust has settled: learning from incidents
22mAfter the dust has settled: learning from incidents
34mReading 100+ Kubernetes Post-Mortems - lessons learned
30mWhat we learned from reading 100+ Kubernetes Post-Mortems
22mAfter Death
44mThe role of catastrophic failure in software design
42mArchitecting a Technical Post Mortem