Finding Needles in the Observability Haystack
summary
A small Python library named opsin library fetches Prometheus/Thanos metrics with caching, converts alert ranges to intervals and Pandas DataFrames, and simulates telemetry so months of data load quickly; it applies one‑hot encoding, dimensionality reduction and clustering to group co‑occurring alerts, includes flapping reduction to merge noisy alerts, and outputs Plotly visualizations. The claim is that simple domain‑aware baselines often beat heavy AI; first release ships with examples and works on synthetic and real clusters.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 8:30Synthetic Data
- 9:13Loading Data
- 10:41Loading Alerts Data
- 11:24Finding Relations Between Alerts
- 12:05Data Transformation
- 12:54Combining Data
- 13:36Identifying Alerts
- 14:19Visualizing Alerts
- 15:07Encoding Alerts
- 16:15Dimensionality Reduction
- 16:38Interactive Visualization
- 17:59Hidden Objects
- 18:23Flappiness
- 20:43What now
- 21:35Wrap up
related talks
37mAIOps: Anomaly Detection with Prometheus and Istio
40mAIOps: Anomaly detection with Prometheus
26mAIOps: New Approaches to Reduce Alert Noise
42mMachine Learning In Observability Systems: From Insights To Action
10mUpstream-First, High Scale Prometheus Ecosystem
24mObservability at Scale: How Open Systems Collects Telemetry from Over 10'000 Edge Devices Worldwide
25mMulti-Cluster Observability: What you need to survive with a fleet
33mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenges