After Death
summary
Explains how DigitalOcean runs postmortems with a 5-point severity scale and concrete timelines (UTC stamps, links to logs and PagerDuty events) and shows that automated alerts and five‑minute Slack heartbeats speed response. Names tools: Prometheus, PagerDuty, Kubernetes, Chef, MySQL, and gives specific failure chains (deploy + missing index + stale worker + aggressive scale). Focuses on system fixes — procedures, testing, decommissioning — not blame, with a shareable postmortem template.
AI-written summary · about
chapters (21)
- 0:00Intro
- 3:00Post-Incident review
- 3:17A system for learning from failure
- 3:27Understand risks to the organisation
- 3:49Being stable is a thing your customers care about
- 5:41A major global outage
- 6:01Incident affecting major product behaviour
- 6:21a minor bug or small defect
- 8:40The incident starts before you notice it
- 9:34Observability
- 9:53Prometheus
- 10:49When did we reach out to a human?
- 12:00When did the human start responding?
- 12:44IR protip: let people know you're doing things
- 13:19How did we fix it?
- 14:48Atlantis deployment ... a scaling factor of 10, where each instance has a default worker thread of 25.
- 16:05Atlantis was issuing unnecessary reads to the production primary
- 16:56Our staging environment does not realistically reflect production operation of our software
- 18:16A lack of procedure around decommissioning chef nodes when services are deployed to kubernetes
- 19:29Procedural, policy, systems and training problems
- 20:00What safeguards were missing?
related talks
22mData-driven Postmortems
51mEmbracing Failure and Learning from Our Mistakes with Effective Post Mortems
30mLearning From Failure: Post Mortems
20mPost Mortem Culture: Learning from Failure
07mDia de los Muertos - Postmortems that save lives
42mArchitecting a Technical Post Mortem
10mSimplify your postmortems and focus on scaling
28mBlameless Postmortem Culture