Improving PySpark Performance Spark performance beyond the JVM
summary
Explains concrete ways to speed PySpark by cutting Python–JVM overhead and shrinking shuffles: uses the py4j bridge, pickling avoidance, RDD cache/persist for reuse, and prefers reduceByKey over groupByKey to avoid huge shuffles (the video reports a measured shuffled read of 48 bytes for groupByKey versus 11 kilobytes for reduceByKey on a laptop). Shows that DataFrames keep data in the JVM, that registering JVM UDFs speeds hot paths, and that toPandas/collect pulls everything to one node.
AI-written summary · about
chapters (19)
- 0:00PySpark performance intro
- 1:27PySpark internals & RDD basics
- 9:31Spark 2.0 changes & perf
- 11:56Python-JVM bridge in PySpark
- 14:45PySpark errors & debugging
- 15:06Word count & RDD reuse
- 17:32Avoid RDD recomputation
- 18:55Key skew handling
- 20:43Why groupByKey is bad
- 24:16Sorting skewed data
- 26:01DataFrames for performance
- 28:28SQL optimizations & pushdown
- 30:35SQL data loading
- 32:21SQL word count & agg
- 34:29Python-JVM mix for perf
- 36:16DataFrames to pandas
- 37:20Joins & key skew pitfalls
- 37:40Near-native UDFs
- 39:07Resources & learning
related talks
43mImproving PySpark Performance: Spark performance beyond the JVM
43mPySpark in Practice
50mBig Data Processing and Data Science Environment in Python
46mInternals of Speeding up PySpark with Arrow
41mKeynote: Making the Big Data ecosystem work together with Python
24mPySpark - Data processing in Python on top of Apache Spark.
45mIndroduction to the PySpark DataFrame API
27mPython and the JVM - A Love Story