Improving PySpark Performance: Spark performance beyond the JVM
summary
Shows that Python Spark suffers from double serialization between Python and the JVM and that the fastest wins come from changing primitives and where work runs: avoid groupByKey and use reduceByKey / aggregate to cut shuffle and memory spikes, persist RDDs when reusing them, move SQL-like steps into DataFrames so work stays in the JVM, and expect Apache Arrow to reduce Python<->JVM costs. Notes Yarn’s default memory split (roughly 90% JVM, 10% Python) and that UDF-heavy pipelines still pay a serialization tax.
AI-written summary · about
chapters (14)
- 0:00Start
- 5:07What is Spark?
- 6:18Spark specific terms in this talk
- 7:15Why people come to Spark
- 8:19Plus a little magic :
- 9:16A detour into PySpark's internals
- 9:35Spark in Scala, how does PySpark work?
- 12:09Everyones favourite wordcount**!
- 14:24What is key skew and why do we care?
- 22:13Word count w/Dataframes
- 22:38Mixing Python & JVM code FTW
- 23:25The "future*": Faster interchange
- 25:29Additional Spark Resources
- 25:40High Performance Spark!
related talks
36mNo more struggles with Apache Spark (PySpark) workloads in production
50mA Java Implementer's Guide to Better Apache Spark Performance
42mScalable Data Science in Python and R on Apache Spark
45mIndroduction to the PySpark DataFrame API
36mScaling data workloads using the best of both worlds: pandas and Spark
11mThe Evolution of Apache Spark™
47mKeeping the fun in Apache Spark Datasets and FP
34mWorking with Tensorflow from the JVM #bbuzz