Detecting Workload Anomalies with Prometheus and Machine Learning
summary
Builds a working Docker Compose lab that wires Telegraf → Prometheus and Grafana → a Python prediction service using Facebook’s Prophet time‑series model, exposing yhat lower/upper plus the forecasted value back into Prometheus. Samples every ~30s, fits configurable change‑points to react faster, and returns triplets (prediction 63, actual 61, bounds 56–70) for each metric. Runs on a laptop, hits CPU/memory limits with longer histories, and recommends delayed alerts (minutes of sustained breach) rather than instant triggers.
AI-written summary · about
chapters (10)
- 0:00Workload anomaly detection intro
- 1:11Monitoring stack setup
- 8:52Telegraf Prometheus Grafana
- 14:52Metrics with Telegraf Grafana
- 30:00ML for anomaly detection
- 34:32Anomaly detection pipeline
- 41:19Metrics exporter for predictions
- 51:58Prometheus Grafana predictions
- 1:05:06Analyzing anomalies tuning
- 1:08:41Future improvements alerting
related talks
05mLightning Talk: Machine Learning Observability with Prometheus
24mObservability in the MLOps Lifecycle with Prometheus
37mLowering the barrier for ML monitoring
37mPractical Capacity Planning using Prometheus
02h 43mPrometheus from A to Z
37mAIOps: Anomaly Detection with Prometheus and Istio
08mPrometheus in 5 minutes
33mTamland: How GitLab.Com Uses Long-Term Monitoring Data For Capacity Forecasting.