Faster PySpark with Apache Arrow

on YouTube ↗29m

" summary

Claims that Apache Arrow reduces PySpark serialization cost and speeds Python UDFs by concrete amounts and shows working examples. Benchmarks show Arrow-optimized Python UDFs ~1.6× faster than pickled UDFs, pandas UDFs ~1.8× faster than Arrow UDFs and ~3× vs regular UDFs and up to ~150× on a SciPy CDF case. Demos include Spark 3.5 Arrow UDF flag, Spark 3.3 mapping Arrow API, forthcoming Spark 4.0 dataframe_to_arrow and a Python data source that reads Arrow record batches via ADBC in ~20 lines of code.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 1:17Arrow and PySpark overview
3. 5:10Python UDFs performance issues
4. 9:29Arrow-optimized Python UDFs
5. 12:19Pandas UDFs for vectorization
6. 15:54Pandas iterator UDFs
7. 17:41PySpark UDF optimizations
8. 19:07Other PySpark Arrow features
9. 23:42Python data source API
10. 25:31Conclusion
11. 26:36Q&A

" related talks

• 46mInternals of Speeding up PySpark with ArrowRuben Berenguel · DataXDay 2020
• 25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new APIDeepak K Gupta · PyCon US 2022
• 31mPolars on SparkAllison Wang, Shujing Yang · PyData Seattle 2025
• 24mPolyglot data with python: Introducing Pandas and Apache ArrowRobson Luis Monteiro Junior · PyCon US 2020
• 33mArrow in Flight - New Developments in Data ConnectivityDavid Li · OSA Con 2022
• 32mThe PyArrow revolution in PandasReuven M. Lerner · PyCon US 2025
• 46mA 101 in Time Series Analytics with Apache Arrow, Pandas and ParquetJay Alexander Clifford · Big Data Conference Europe 2023
• 41mFaster SQL with pandas and Apache ArrowWill Ayd · PyData NYC 2023