Apache Spark Performance Tuning Session with Delight
summary
Delight is an open-source Spark monitoring dashboard that injects a driver agent jar, streams encrypted metrics to a hosted backend, and aligns memory, CPU and IO graphs with jobs and stages so you can pinpoint root causes. It reports cpu uptime, task duration and efficiency (examples: 20% on a skewed job, 86% on a healthy run). Real fixes shown: switching from EBS to local SSD cuts a shuffle-heavy pipeline from >2.5 hours to 26 minutes. Setup works on Databricks, EMR and Kubernetes; data appears a few minutes after job end.
AI-written summary · about
chapters (4)
- 0:00Start
- 1:55Apache Spark Is the N1 Distributed-analytics Engine for Big Data and Al
- 2:40But Managing a Spark Infrastructure and Debugging Spark Is Hard
- 8:37Delight Consists of an Open-source Agent Talking to a Hosted Backend
related talks
30mOptimize Spark Jobs Using Its UI
32mScalable Monitoring Using Prometheus with Apache Spark
34mAdvanced visualization of Flink and Spark jobs
40mProductionizing Big Data - Stories from the Trenches
50mA Java Implementer's Guide to Better Apache Spark Performance
36mNo more struggles with Apache Spark (PySpark) workloads in production
24mTaming Data Skew in Production ML Pipelines
30mSpeed Up Your Spark Data Processing with a GPU