Anomaly Detection for Cloud Native Storage
summary
A telemetry pipeline collects volume metrics—IOPS, throughput, latency, capacity—from storage backends and node exporters, exports them via Prometheus exporter, stores about 6,000 datapoints from short experiments and pulls them into an ML module. The ML module runs DBSCAN and Gaussian models (also testing kNN/clustering), flags outliers seen in MongoDB and Ceph testbeds, and sends alerts through Prometheus Alertmanager or a REST hook. The system works end-to-end for detection but remains early stage, focused on gathering more data and adding root cause analysis.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:17What is Anomaly Detection
- 2:52Anomaly Detection Use Cases
- 4:04Storage Performance Challenges
- 5:06Yahoo! JAPAN's Environment
- 5:42Storage for Private Cloud
- 6:40Future environment we hope
- 7:08Software Defined Storage
- 7:52The Open Autonomous Data Platform
- 8:33The Road To Autonomous Data Platform
- 10:20Collecting Storage Performance Metrics
- 10:59Prometheus Architecture
- 12:57Metrics interface
- 13:39Collect LVM Metrics
- 14:06Collect Ceph Metrics
- 14:43LVM Metrics in Grafana
- 15:26Emit Metrics to Prometheus (recap)
- 15:39Prometheus Alert Manager
- 16:35Anomaly Detection Architecture
- 17:44Anomaly Detection Algorithms
- 19:08DBSCAN Clustering
- 20:15Gaussian Model Graph - LVM
- 22:40DBSCAN Graph - Ceph
- 24:29DBSCAN Graph - Node Exporter
- 30:09What's Next
related talks
37mAIOps: Anomaly Detection with Prometheus and Istio
40mAIOps: Anomaly detection with Prometheus
26mAnomaly Detection System for the CERN Cloud
38mWatch Everything, Watch Anything: Anomaly Detection
21mAnomaly Detection for Platform Logs using Embedded Context
36mObservability with Prometheus and Beyond: Leveraging Cloud-native Technologies Outside of Cloud
30mPractical Anomaly Detection at Scale With PromQL
34mML-Bound Anomaly Detection: Experiences Beyond Commodity Observability