Modeling Alert Quality
summary
Defines alert quality as three concrete classes—true alarms, false alarms, and missing alarms—and models it with measurable intervals: start-to-detect, detect-to-acknowledge, diagnosis time (when the correct fix is known), and remediation time. Claims false and useless alerts are pure cost and good alerts must reduce total incident cost (engineer hours, disruption, SLA and reputational loss). Recommends instrumenting hours worked and harm per minute, tracking dominant terms separately because overall metrics are noisy and laggy, and warns against using quality as an OKR.
AI-written summary · about
chapters (27)
- 0:00Intro
- 0:34Acknowledgement of Country
- 0:46What are alerts?
- 1:03Monitoring
- 1:50Low priority alert
- 2:07High priority alert
- 2:42What is alert quality made of?
- 6:04True Alarm
- 10:35Missing Alarm
- 15:00Non-alerting costs
- 17:13Alert quality as value
- 17:36Breaking down costs
- 18:23Cost of false alarm
- 19:01Incident cost
- 19:38Remediation cost: disruption
- 20:18Remediation cost: Work involved
- 20:31Incident loss
- 21:20Time to detect
- 21:34Time to acknowledge
- 22:16Reputation harm
- 23:00Measuring value
- 23:41Gather data
- 25:47Tracking quality: black swans
- 26:46Tracking quality: Goodhart's law
- 27:41Summary: Alert quality matters
- 27:54Summary: Alert quality difficult to track
- 28:11Summary: Alert improvement
related talks
06mIt’s Just an Alert, Don’t Wake Up!
25mSymptom-based Alerting for Machine Learning - What I Learned...
28mAlerts don't suck, YOUR alerts suck
36mThe road to world-class observability
27mError Culture with Ryan Cheley
32mWho Are Your Alerts For?
31mAlerts Don't Suck YOUR Alerts Suck!
19mSensory Friendly Monitoring: Keeping the Noise Down