Alert Management as Code
summary
Describes building an alert-as-code pipeline that integrates Zabbix and Prometheus with Grafana, Alertmanager and the ELK stack, stores alert metadata in version-controlled files, and applies changes via Terraform-managed alerts. Adds automated layers called Autobot and Coroner that open incidents and run diagnostics, reducing alert noise, improving prioritization by user impact, and shortening alert delivery from weeks to minutes while remaining reproducible in production.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 5:20Monitoring terms
- 8:55Zabbix Prometheus components
- 12:42Monitoring integration issues
- 15:32Dev teams incident roles
- 17:18Alert noise consequences
- 20:53Fixing monitoring errors
- 23:45App systematization
- 26:18Alert priorities
- 28:28Alerts Continuous Delivery
- 31:19Ineffective alerts
- 34:55Incident management
- 35:40Monitoring improvements
- 37:46Feedback culture
- 41:11Q&A
- 42:56Business monitoring Q&A
- 44:25Alert deduplication Q&A
- 46:00ML in monitoring Q&A
related talks
17mMonitoring as Code
18mAlert Analytics
50mMetrics and alerting with Prometheus and Grafana
23mAlerting in the Prometheus Ecosystem: The Past, Present and Future
33mAlerting in Cloud Native Environments
26mAutomating Kibana Alerting: When GitOps meets UI
50mApplied Alerting Philosophy
37mSmart Data-driven Alerts (SDdA) and How They Make DevOps Life Easier