AI Driven Observability based on Open Source
summary
An open‑source Data Metric Analyzer ingests millions of metrics per minute from ~30k–40k servers using Telegraf → TimescaleDB → Airflow, trains monthly models from three months of history, and flags anomalies to avoid cloud overspend; initial RNNs give best accuracy but are CPU‑heavy so production uses XGBoost over RNN for trend forecasting and anomaly scoring. A Postgres case reclaims 5.4TB of 8.5TB via unused pages, and the stack runs on Kubernetes with Streamlit/Grafana visualizations.
AI-written summary · about
chapters (7)
- 0:00Introduction and audience questions
- 3:30Observability pain points explained
- 6:45Cloud database challenges
- 12:00Proactive anomaly detection
- 15:30Data science approach to monitoring
- 22:20Open source reference architecture
- 25:14Q&A session
related talks
33mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenges
22mAI with Machine Learning for Observability in Netdata
22mAI Observability At Meta Scale
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
41mCustomer Centric Observability at Scale Leveraging AIOps & OpenTelemetry
41mScaling Observability: From Chaos to Clarity with Open Source Tools
26mAI-Driven Cloud Observability: Reduce toil and amplify time to resolution
40mOptimizing AI Applications for Production with Observability