A Custom Incident Notification System
summary
A team built an in-house alerting system for emergency situations that decides who to wake up at night. The presentation explains handling noisy or missing metrics across multiple services (when to alert all admins vs. suspect monitoring failure), and describes their approach and lessons learned while designing this notification logic and system.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 7:46First design choice
- 11:37Parallel metrics stream
- 15:30Expert systems thresholds
- 17:20Titan-2 rocket story
- 20:29Single point of failure
- 23:00Manual work API
- 26:54API quality changes
- 27:14Nuclear plant story
- 28:39Multiple notifications problem
- 31:059-to-5 work
- 35:00Sound notifications innovations
- 38:11Q&A
related talks
19mSensory Friendly Monitoring: Keeping the Noise Down
32mWho Are Your Alerts For?
50mApplied Alerting Philosophy
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
19mSensory Friendly Monitoring
45mIncident Management - Talk the Talk, Walk the Walk
19mSensory Friendly Monitoring: Keeping the Noise Down
27mBuilding a Scalable Monitoring System