Alerting in Cloud Native Environments
summary
Defines alerts directly on Prometheus time series and shows concrete rules: compute per-second rates from counters, divide errors by requests to alert on error rate >1% instead of raw counts (avoids noisy >500/sec thresholds), preserve or drop labels to avoid signal cancellation, use linear regression on free bytes to predict disk full within 4 hours, and route/deduplicate notifications through alertmanager with grouping, silences and inhibitions. Examples include multi‑instance aggregation, label-based routing, and practical safeguards so alerts stay actionable rather than noisy.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 0:22Timeseries
- 2:01Prometheus
- 2:53Big Data
- 3:25Monitoring at Scale
- 4:59SLA
- 7:29Causebased alerting
- 9:07Prometheus DSL
- 10:35Detecting Conditions
- 11:49Time Series
- 12:24Error Rates
- 14:50Signal Cancellation
- 15:44Preserving Dimensions
- 16:18Instance Label
- 17:08Query Language
- 18:57Capacity Situation
- 21:27annotations
- 22:11alert management
- 25:15silencing
- 25:28inhibition
- 30:26depth
related talks
18mAlert Analytics
50mApplied Alerting Philosophy
23mAlerting in the Prometheus Ecosystem: The Past, Present and Future
17mMonitoring as Code
50mMetrics and alerting with Prometheus and Grafana
46mCluster Golden Signals to Avoid Alert Fatigue at Scale
08mPrometheus in 5 minutes
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise