ML-Bound Anomaly Detection: Experiences Beyond Commodity Observability
summary
Explains using ELK (Elastic Stack with Kibana) anomaly-detection (ML nodes running unsupervised jobs) to replace static thresholds, with a concrete single-metric job that watches log-entry rate and produces anomaly score 35 when it detects a sixfold increase in messages. Describes model internals (time-bucketed probability scores, detrending, categorization), operational outcomes (alerts via Slack/Email/PagerDuty, potential 15% CIE reduction), and practical tradeoffs: false positives, noisy inputs, and extra ML-node compute cost.
AI-written summary · about
chapters (15)
- 0:00Introduction and agenda
- 1:05Cloud native adoption trends
- 4:36Microservices architecture benefits
- 6:47Challenges in scalable systems
- 8:52Importance of robust monitoring
- 10:21Machine learning for alerting
- 12:07ELK stack overview
- 13:54What is anomaly detection
- 16:25Elasticsearch anomaly workflow
- 19:04How anomaly detection works
- 23:00ML model capabilities
- 26:12ML job types and features
- 28:20Demo anomaly detection results
- 31:14Pros and cons summary
- 33:44Conclusion and Q&A
related talks
30mBetter data quality with ML & visualization
10mML-based Log Analysis for Faster Debugging
31mAnomaly Detection on OpenStack Logs using M.L.
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
25mAutomating Operations with ML
25mSymptom-based Alerting for Machine Learning - What I Learned...
41mScalable Anomaly Detection (with Zero Machine Learning)
28mML-Driven Test Intelligence at Scale - What Works What Fails and Why It Matters