Alert Analytics
summary
Cloudflare runs over 1,000 Prometheus servers across 285 PoPs and routes alerts into a pipeline that uses vector.dev pipeline to ingest webhooks and poll Alertmanager, stores events and silences in ClickHouse store, and visualizes noise with a Grafana swimlane view. The system finds silent firing alerts, decommissioned-cluster noise, and frequent flapping; fixes include adding notify labels, adjusting Prometheus for_loss and thresholds, using Alertmanager inhibition and grouping, and a Docker Compose demo repo that reproduces the pipeline and dashboards.
AI-written summary · about
chapters (15)
- 0:00Alert fatigue introduction
- 1:28On-call health impacts
- 2:54Alert analytics importance
- 4:19Alerting stack overview
- 5:44Alertmanager core concepts
- 7:14Alert reporting challenges
- 9:02Vector.dev for pipelines
- 10:05ClickHouse alert storage
- 11:31Dashboards for alert analysis
- 12:36Alert cleanup findings
- 13:39Actionable alert creation
- 14:45On-call workload visualization
- 15:48Flapping alerts causes
- 16:29Alert management best practices
- 16:53Live demo and repo
related talks
33mAlerting in Cloud Native Environments
23mAlerting in the Prometheus Ecosystem: The Past, Present and Future
50mMetrics and alerting with Prometheus and Grafana
50mApplied Alerting Philosophy
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
48mAlert Management as Code
46mCluster Golden Signals to Avoid Alert Fatigue at Scale
08mPrometheus in 5 minutes