Monitoring as Code
summary
Argues that alert rules live in source control and are deployed like app code, using Prometheus expressions (example: my_failures_total increases by more than 5 in 4 minutes triggers an alert) and unit tests that generate fake time series with promtool unit tests. Uses Kubernetes configMaps for Prometheus rules with a configmap-reload sidecar, and manages external checks via the terraform provider Pingdom with CI running terraform plan on PRs; notes practical fixes like a terraform-version wrapper and a 10,000 goroutines leak alert.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:22Overview
- 0:58Why Monitoring
- 2:30Prometheus
- 8:55Deployments
- 10:05Other Monitoring Systems
- 10:45API
- 11:15Terraform
- 12:20Source Graph
- 14:14Questions
related talks
04mLightning Talk: Prometheus Rules management and validation
33mAlerting in Cloud Native Environments
41mMonitoring Like a Boss
32mTesting Kubernetes apps' observability end-to-end
48mAlert Management as Code
50mMetrics and alerting with Prometheus and Grafana
13mWatching the Watchers: Generating Absent Alerts for Prometheus
27mMonitoring as Code: Managing your Observability Assets the Cloud-Native Way