Netdata: Open Source, Distributed Observability Pipeline – Journey and Challenges
summary
Open-source observability built to collect 3,000 metrics per second on a VM (versus ~100 metrics every 15s in DataDog), store samples at ~0.5 byte/sample with Gorilla+zstd and no per-sample timestamps, and run at edge agents that use ~1% CPU and ~120 MB RAM with ~3 GB disk for two years retention. The system auto-discovers everything, auto-generates dashboards via the Needle metadata model, ships alert packs (≈350 alerts), beats Prometheus in tests (1/3 CPU, half memory, 10% less bandwidth), integrates systemd journal with a log-to-journal tool, and runs 18 ML models per time series trained on ~2.5 days of history for real-time anomaly scoring.
AI-written summary · about
chapters (10)
- 0:00Introduction and motivation
- 3:57Bad best practices in monitoring
- 8:14High-fidelity data collection
- 12:15Distributed observability pipeline
- 17:20Open source and CNCF status
- 20:34Fully automated dashboards
- 23:00Real-time anomaly detection
- 27:21Logs and systemd Journal integration
- 30:33Advanced observability features
- 31:58Netdata Cloud and features
related talks
44mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenge
29mAI Driven Observability based on Open Source
41mScaling Observability: From Chaos to Clarity with Open Source Tools
24mNew approach to metrics observability for the cloud era
52mReal-time-first metrics
24mObservability at Scale: How Open Systems Collects Telemetry from Over 10'000 Edge Devices Worldwide
08mNetwork Performance Monitoring
31mThe Observability Data Lake, 1 year on