Who Are Your Alerts For?
summary
Built a custom alerting layer on Influx that stages new alerts in git for a week and enforces fine‑tuning, with per-alert frequency knobs and a subscription model so engineers own only alerts they care about. Alerts that say 9.2 is above 9 or emit only critical severity create confusion, hours‑long mean time to respond, and burnout; the staged testing plus automation for unattended alerts reduces noise, raises trust, and makes human responders faster and less stressed.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 0:55What are Alerts
- 2:36What is this talk about
- 3:35Takeaways
- 4:11Alerts
- 5:17Challenges
- 6:06You know data
- 6:58Uncertainty
- 8:53Analytical Challenges
- 10:35Why Do We Care
- 10:54Why You Should Care
- 12:52What should you not do
- 13:15Example
- 15:00Consequences
- 15:47Alert Rule 1
- 16:49Alert Rule 2
- 18:03Level Critical
- 19:40Critical Events
- 20:30Too Many Alerts
- 20:48Alert Fatigue
- 22:02Customization
- 24:43I dont care
- 25:35Why alerts are bad
- 26:25Subscription model
- 28:00Cultural transition
- 28:14Engineers must build trust
- 29:26Biggest pitfalls
- 30:14Building trust
related talks
33mTackling Alert Fatigue with SLOs
50mApplied Alerting Philosophy
40mNext-generation Alerting and Fault Detection
33mAlerting in Cloud Native Environments
28mAlerts don't suck, YOUR alerts suck
27mError Culture with Ryan Cheley
29mDraining the Flood—A Combat against Alert Fatigue
19mSensory Friendly Monitoring: Keeping the Noise Down