Embracing Failure and Learning from Our Mistakes with Effective Post Mortems

on YouTube ↗51m

" summary

Describes how Datadog runs blameless postmortems end-to-end: collect time-series metrics (they ingest ~1 trillion points/day), capture Slack control-call logs, build a timeline annotated with graphs, and run monthly company reviews. Recommends classifying signals as work/resource/event metrics, keeping fine-grained, tagged data long enough to investigate, and tracking action items in GitHub/Trello. Emphasizes quick notes after incidents, avoid blame, prioritize customer-impact fixes, and add tooling checks (e.g., command validation) that actually prevent repeat failures.

AI-written summary · about

" chapters (10)

1. 0:00Embracing Failure and Learning from Our Mistakes with Effective Post Mortems llan Rabinovitch, Datadog
2. 2:58THE PROBLEMS WE WORK ON AT DATALOG ARE HARD AND OFTEN DON'T HAVE OBVIOUS. CLEAN- CUT SOLUTIONS. SO IT'S USEFUL TO CULTIVATE YOUR TROUBLESHOOTING SKILLS, NO MATTER WHAT ROLE YOU WORK IN.
3. 17:00RESOURCE METRICS UTILIZATION SATURATION ERROR AVAILABILITY
4. 18:454 QUALITIES OF GOOD METRICS Well-understood Granular Tagged by scope Long lived
5. 24:41TECHNICAL ISSUES HAVE NON-TECHNICAL CAUSES HUMAN ELEMENT
6. 28:08"ONE PICTURE IS WORTH TEN THOUSAND WORDS" CHINESE PROVERB
7. 34:53SUMMARY: WHAT HAPPENED? Describe what happened here at a high-level- think of it as an abstract in a scientific paper. What was the impact on customers? What was the severity of the outage? What components were affected? What ultimately resolved the outage?
8. 40:34WHY DID IT HAPPEN? Deep dive into the cause Examples from this incident
9. 41:45HOW DO WE PREVENT IT IN THE FUTURE? Link to Github issues and Trello cards Now? Next? Later? Follow up notes
10. 44:14RECAP: What happened summaryl? How did we detect it? How did we respond? Why did it happen deep dive/? Actionable next steps!

" related talks

• 22mData-driven PostmortemsJason Yee · ChefConf 2018
• 30mLearning From Failure: Post MortemsAlex Gaynor · PyCon 2018
• 28mBlameless Postmortem CulturePranjal Deo · Conf42 Chaos Engineering 2021
• 20mPost Mortem Culture: Learning from FailureYury Yineth Niño Roa · Conf42 Chaos Engineering 2020
• 31mThe blameless post mortem: how embracing failure makes us betterChris Wilcox · DjangoCon US 2019
• 23mWho to blame for all your problemsBen Berry · PyCascades 2019
• 10mSimplify your postmortems and focus on scalingRicardo Aravena · LeadDev New York 2022
• 22mAfter DeathSam Phippen · GORUCO 2018