Building a Large-Scale, Streaming-Based Logging and Monitoring Solution
summary
A streaming pipeline based on Riemann ingests heterogeneous sources and handles more than 5,000 events/sec, writing terabytes into Elasticsearch and metrics into Prometheus, with Kibana/Grafana for views and Alertmanager plus a custom smart alarms GUI. A functional core (ports-and-adapters) exposes a small domain specific language for event matching, runs about 35 containers managed with Nix and Docker Compose, uses self‑healing breakers when outputs stall, and plans inverted indices to speed matching.
AI-written summary · about
chapters (7)
- 0:00Start
- 3:40Pandemonium
- 7:26Input Sources
- 8:51Output Destinations
- 11:47Functional architecture
- 14:41Functional Core
- 26:32Lessons learned, future work
related talks
46mBuilding a scalable logging platform
42mMake your logs work for you
30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in Production
40mA needle in a logstack
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale
41mScaling Telemetry Systems with Streaming
29m“If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus
31mThe Long and Winding Road to Application Logging [A]