Netdata: Open Source, Distributed Observability Pipeline – Journey and Challenge
summary
Netdata is an open‑source observability pipeline that keeps full‑resolution data at the edge with edge-first agents and optional parent clusters plus a non‑storing cloud proxy. It collects per‑second metrics, runs lightweight on‑node ML (unsupervised k‑means) for per-second high‑fidelity anomaly detection, stores anomaly bits, and ranks signals with a k‑means anomaly scoring and an anomaly advisor for root cause. Claims: 50–90% lower observability cost and a 2.7M metrics/s stress test showing much lower CPU, ~50% memory, and ~98% less disk than Prometheus.
AI-written summary · about
chapters (12)
- 0:00Introduction and overview
- 1:47Netdata’s unique design
- 4:59Centralization problems in observability
- 6:53Edge-based observability pipeline
- 10:29Agent efficiency and performance
- 14:24Five challenges and solutions
- 20:43Real-time data fidelity
- 23:13Machine learning at the edge
- 28:57Logs and systemd Journal
- 31:29Netdata Cloud and monetization
- 35:05Q&A: Distributed architecture
- 40:04Q&A: Windows and Kubernetes support
related talks
33mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenges
06mIGNITE: Practical AI with Machine Learning for Observability
22mAI with Machine Learning for Observability in Netdata
08mNetwork Performance Monitoring
52mReal-time-first metrics
40mBuilding a Scalable Observability Platform for Edge Compute Nodes
29mData Observability: From 1788 To 2032
35mAnomaly Detection for Cloud Native Storage