Applied Alerting Philosophy
summary
Describes applying Rob Ewaschuk’s symptom‑based alerting with concrete tooling: metric instrumentation, HAProxy exporter, Prometheus alertmanager rules that compute SLO error rates and multi-window thresholds (five‑minute plus one‑hour with burn factors like 6 and 14.4). SoundCloud builds automated routing from labels to owners and uses the multi-window, multi-burn-rate alerts to page only on imminent SLO burn while tickets handle slower issues — implemented in production and effective at cutting noisy pages.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 0:27Prometheus
- 0:51SoundCloud
- 1:08Clone
- 1:25Dashboard
- 1:51Applied Alerting Philosophy
- 5:53Service Level Objectives
- 6:13Practical Alerting
- 6:55Open Source Monitoring Systems
- 7:35Site Reliability Workbook
- 9:40Symptoms vs Courses
- 13:19Pages vs Tickets
- 17:26Symptoms
- 20:24Blackbox probes
- 22:49Prometheus blackbox probes
- 27:10One person symptom
- 28:10Application
- 34:01Alerting on SSL
- 34:40MultiBurn
- 36:10PromptQL
- 37:12Not Valid Concern
- 38:51Timeline
- 42:04Budget Burn
- 42:34Duration
- 44:18Page
- 44:47System Label
- 45:29Alert Manager
- 46:07Questions
related talks
33mAlerting in Cloud Native Environments
41mA Theory and Practice of Alerting with Service Level Objectives
40mNext-generation Alerting and Fault Detection
39mActionable Observability
18mAlert Analytics
36mHow to Include Latency in SLO-based Alerting
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
19mSensory Friendly Monitoring: Keeping the Noise Down