Test Driven Alerting: Now with More Data Science!
summary
Claims a test-driven framework that joins simple data science with Prometheus to cut alert noise and enable automation. Uses Prometheus functions like holt_winters and predict_linear plus thresholds, traffic minimums and alert duration (10 minutes) to eliminate false positives. Cites big data scale — engines with ~5,000 sensors producing up to 10 GB/s (≈844 terabytes per transatlantic flight) — to justify automation. Implements routing via alertmanager grouping, treats self-healing as filterable, and considers further ML methods; prototype-level, not a finished product.
AI-written summary · about
chapters (11)
- 0:00Introduction and speakers
- 1:05Pagers and alerting pain
- 5:06Data deluge and IoT insights
- 7:13Cloud Foundry monitoring challenges
- 14:01Rise of data science
- 18:43Data science basics for alerting
- 20:06Reducing alert noise with Prometheus
- 23:33Smart filtering and routing
- 27:31Test-driven alerting vision
- 29:16Call to action and summary
- 30:41Q&A session
related talks
37mSmart Data-driven Alerts (SDdA) and How They Make DevOps Life Easier
26mAIOps: New Approaches to Reduce Alert Noise
17mMonitoring as Code
41mMetrics Applied
48mAlert Management as Code
20mBuilding a Data Pipeline with Testing in Mind
40mThe Prometheus monitoring system
23mMastering Monitoring & Alerting: Key to Seamless Performance