AIOps: Anomaly Detection with Prometheus and Istio
summary
Pipeline ingests Prometheus metrics (including Istio instrumentation), persists them to object storage or Prometheus → Thanos long-term storage, and loads them into Apache Spark and Jupyter notebooks for analysis. Forecasting uses Facebook Prophet forecasts (with hourly retrain) and occasional Fourier models; a simple accumulator flags deviations and the system exposes results as a Prometheus endpoint so alerts can use Predictions exposed via /metrics. Prototype handles ~360,000 metrics/hour; Prophet sometimes misses spikes so Fourier or tuned thresholds improve detection, online training is not yet implemented.
AI-written summary · about
chapters (5)
- 0:00Start
- 7:57Prometheus architecture
- 16:35Prometheus Metric Types
- 18:56Monitoring is hard
- 26:24Runtime configuration
related talks
37mAIOps: Anomaly Detection with Prometheus and Istio
40mAIOps: Anomaly detection with Prometheus
15mScaling Observability Stack for AIOPS
24mObservability in the MLOps Lifecycle with Prometheus
35mAnomaly Detection for Cloud Native Storage
01h 12mDetecting Workload Anomalies with Prometheus and Machine Learning
01h 21mBuilding a ML Model Observability pipeline
05mLightning Talk: Machine Learning Observability with Prometheus