Keynote: Making the Big Data ecosystem work together with Python
summary
Shows concrete fixes: JVM↔Python double-serialization via Py4J and custom pickling causes huge overhead and copying; replacing that path with Apache Arrow and using pandas UDFs in Spark 2.3 moves columnar data instead of individual pickles, yielding typical speedups around 1.5× (despite a shouted 242x claim), enables grouped-map pandas operations, spaCy calls from the JVM, and a prototype sparkling-ml bridge; demos and GitHub code exist and Docker is used for shipping consistent worker environments.
AI-written summary · about
chapters (14)
- 0:00Welcome
- 2:57Big data interoperability challenges
- 6:58Python-JVM data sharing
- 9:30Spark-Python performance
- 13:06Double serialization costs
- 16:56Reducing data copying
- 19:28Apache Arrow for exchange
- 22:01Other systems and Python
- 24:11Dask dependency management
- 26:47Python-JVM integration trends
- 28:34Multi-language pipelines demo
- 33:58SpaCy and Spark use case
- 38:01Resources
- 39:31Q&A
related talks
50mBig Data Processing and Data Science Environment in Python
14mKeynote: Scalable Python We Can Afford
31mBringing the Jewels of the Python World to Scala with Spark
24mPolyglot data with python: Introducing Pandas and Apache Arrow
41mImproving PySpark Performance Spark performance beyond the JVM
46mInternals of Speeding up PySpark with Arrow
44mUsing Python with Hadoop to create production ready big data applications
43mPySpark in Practice