Optimize Spark Jobs Using Its UI
summary
Shows how to debug and optimize Apache Spark jobs using the Spark UI, History Server event logs and the Spark 3 JSON API when source code is unavailable. Demonstrates concrete checks (executor GC/memory, SQL tab physical plan, stage/task skew and task durations), reproduces an out-of-memory case where a single task processes most data, and explains using stage IDs and dataframe field tags to confirm skew and build a reconstruction graph. Recommends avoiding blind repartition and instead using a bucketing key or Spark adaptive execution to fix skew.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:46History Server
- 9:35Spark UI overview
- 18:26Failed stages analysis
- 24:10Optimization via Spark History API
- 29:04Q&A
related talks
43mOptimizing Spark applications from simple to complex with examples
34mAdvanced visualization of Flink and Spark jobs
29mApache Spark Performance Tuning Session with Delight
15mOptimizing Query Performance in Big Data Systems
21mAvoiding Spark Pitfalls at Scale
36mNo more struggles with Apache Spark (PySpark) workloads in production
24mTaming Data Skew in Production ML Pipelines
24mImproving interactive querying experience on Spark SQL