Monitoring Big Data Systems Done "The Simple Way"
summary
Argues that big data monitoring can stay simple by baking checks into the dev pipeline and using open source stacks: Spark 10–300 nodes runs analytics, data lands on S3/Cassandra/MongoDB, metrics go to Graphite or InfluxDB and Grafana, logs to ELK/Kibana, and alerts route to Slack or PagerDuty. Uses Jenkins + JUnit checks to verify data flow and counts, ships metrics from apps, and adds ML log anomaly detection via CoreLogics; claims two people part‑time can operate it if monitoring is automated into development.
AI-written summary · about
chapters (23)
- 0:00Start
- 4:36What is Big Data (IMHO)? And What to Monitor?
- 5:59Monitor What?
- 7:18Monolith Structure
- 8:04Distributed Microservices Architecture
- 8:53Basic Concepts
- 10:33Structure of the Data
- 13:30Monitoring Stack - Let's fill in the blanks
- 14:17Situations & Problems
- 14:39Cassandra Spark
- 15:33Monitoring Operation System Metrics
- 16:11Google Cloud Platform StackDriver
- 16:49Amazon Web Services CloudWatch
- 17:02Report to Where?
- 18:12Drivers to Datastores
- 20:04Monitoring Cassandra
- 21:39What to monitor in an Apache Spark Cluster
- 22:12Ways to Monitoring Spark
- 22:50Data Questions? What should be measure
- 24:00Data Answers!
- 29:56Alerting
- 30:20Summary - Monitoring Stack
- 31:58Conclusions
related talks
05mApply Small Data to Big Data
33mAccidental DataOps: How we matured our organization with data, and how you can too!
33mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenges
01h 15mMonitoring: Turning Noise into Signal
29mDIY Monitoring with Open Source Analytics
21mMonitoring Your Apps is not About Devops
30mArchitecting Applications With Multiple Open-Source Big Data Technologies
47mMonitoring at Scale: Intuitive Dashboard Design