Practical Anomaly Detection at Scale With PromQL
summary
Grafana Labs builds an anomaly system implemented entirely in PromQL recording rules that runs at scale in production and is being rolled out to customers. It creates per-service baselines for USE/RED and tagged metrics using a 1-hour midline / 26-hour stddev pattern, multiplies stddev by 2, uses a coefficient‑of‑variation threshold (~50% of the mean) as a high‑pass filter, combines short/long/seasonal bands, exposes alerts as events (not paged) and ships the rules and dashboards open source with an asserts_anomaly="gauge" label.
AI-written summary · about
chapters (9)
- 0:00Introduction to anomaly detection
- 3:36Building the basic model
- 7:33Handling extreme outliers
- 9:53High pass filtering technique
- 12:14Addressing discontinuities
- 13:20Incorporating seasonality
- 14:24Implementing the framework
- 17:43Troubleshooting with anomalies
- 20:57Q&A and conclusion
related talks
06mEasy anomaly Detection with PromQL
29mAlerting and Anomaly Detection – Best Friends Forever?
50mMetrics and alerting with Prometheus and Grafana
35mAnomaly Detection for Cloud Native Storage
31mFirst Day Foresight: Anomaly Detection for Observability
38mWatch Everything, Watch Anything: Anomaly Detection
46mHow many dashboards does your Grafana have?
34mAnomaly Detection in Time Series from Scratch Using...