Data Driven Observability & Incident Response
summary
Shows how PagerDuty and Humio power a real-time, data-driven observability pipeline with Humio ingest and search latency reported at less than one second, enabling live investigation and faster fixes. Describes building regular failure injection (they run “Failure Fridays”) and using an Incident Command System with a rotating incident commander (often a product owner) to coordinate responders. Cites Netflix’s experiment of taking down 50% of production servers with zero user-visible impact, and advocates blameless postmortems and proactive programming despite up-front time cost.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:24About Matty
- 1:29About PJ
- 2:35Definitions
- 5:57Feedback Loop
- 9:40Observability
- 16:32Proactive Programming
- 21:51All of it Together
- 23:08What is incident response?
- 26:51Don't litigate severity
- 28:47Incident Commander
- 29:51Not a resolver
- 31:22Highest Ranking
- 32:47Executive Swoop
- 34:43Notify stakeholders
- 35:35Sizing up
- 37:16Making decisions
- 39:03Rally fast, disband faster
- 41:11Don't panic
- 44:19At the end of the call
- 45:13Post-mortems
- 47:32How do we do it?
related talks
45mHow to Infect your Organization with Humane Ops
23mMotivational Leadership Driving Change Through Empowered Eng
30mHow HashiCorp does Incident Management
05mKeynote: Observability in the Next Decade, Beyond Reactive Response
39mHuman Observability of Incident Response
42mPlan for Unplanned Work: Game Days with Chaos Engineering
28mRevolutionizing Incident Response
05mDevOps and incident management: A recipe for success