Scalable Monitoring Using Prometheus with Apache Spark
summary
Shows a reproducible OpenShift/Kubernetes setup that instruments Spark with a Java JMX agent and configures Prometheus targets via a Spark operator CRD, stores metrics in Prometheus and visualizes them with Prometheus + Grafana dashboards. Demonstrates a concrete optimization: caching an RDD cuts per-worker heap from ~5.3 GB to ~2 GB (cached RDD reduced worker memory from 5.3GB to ~2GB (≈76%)), and yields about a 10% runtime decrease, with alerting via Alertmanager and configurable scrape intervals.
AI-written summary · about
chapters (7)
- 0:00Introduction to speakers
- 1:03Project overview
- 3:52Concepts of Spark and Prometheus
- 7:42Architecture and configuration
- 13:43Performance analysis strategy
- 17:28Memory management and tuning
- 23:53Live demo and conclusion
related talks
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes
29mDynamic Large Scale Spark on Kubernetes
37mPractical Capacity Planning using Prometheus
52mSolving Analytical Problems using Apache Spark
21mAvoiding Spark Pitfalls at Scale
28mCloud-Native Apache Spark: why & how to migrate your Spark
29mApache Spark Performance Tuning Session with Delight