Spark Performance Tuning and Troubleshooting
summary
Shows concrete fixes and measured wins: a UDF that makes an HTTPS call is slow because of connection-per-record and becomes fast when a single connection is reused per task cutting SSL handshake overhead (example drops from ~100s to ~20s). Demonstrates replacing repeated groupbys with one aggregation to shrink a DAG from 46 stages to 8 stages and drop runtime from ~20 minutes to ~3 minutes. Covers using Spark profiler, broadcast joins, native vs Python UDFs, partition-level mapping, determinism issues with sampling and adaptive execution.
AI-written summary · about
chapters (7)
- 0:00Spark performance tuning
- 1:29Data skew in Spark jobs
- 2:34Optimizing UDFs with APIs
- 8:45Wide data with many columns
- 19:48Deterministic ML pipelines
- 29:49Spark optimization key points
- 30:59Q&A on UDFs
related talks
41mOptimising Apache Spark and SQL for Improved Performance
25mData Demons: Unleashing Chaos or Driving Performance?
47mApache Spark Lakehouse Pushdowns with Transpilation
15mOptimizing Query Performance in Big Data Systems
40mScaling with Apache Spark (or a lesson in unintended consequences)
50mChallenges that everyone struggles while productionizing Apache Spark workloads
43mBeyond Shuffling: Scaling Apache Spark
29mFaster PySpark with Apache Arrow