The Evolution of Apache Spark™

on YouTube ↗11m

" summary

Apache Spark now treats Python as a first-class language with API, error-message and performance work; PySpark downloads span 200+ countries, and Databricks customers run >5 billion PySpark queries per day on Spark 3.3+. Dependency and upgrade pain from the single-process driver motivate Spark Connect (gRPC + Apache Arrow), which decouples client applications, enables language-agnostic bindings (Go, Rust, etc.), isolates debugging, and lets server and apps upgrade independently.

AI-written summary · about

" chapters (5)

1. 0:00Apache Spark evolution
2. 1:26Python support
3. 6:45Spark Connect and deps
4. 10:00Sparklyr and R
5. 11:06Q&A

" related talks

• 55mUse Spark from AnywhereDenny Lee, Ginger Holt · Scala Days 2023 Seattle
• 24mPySpark - Data processing in Python on top of Apache Spark.Peter Hoffmann · EuroPython 2015
• 39mHow Python Drives Modern Data EngineeringKiana Alessandra Villaera · PyCon Philippines 2023
• 30mImproving PySpark Performance: Spark performance beyond the JVMHolden Karau · PyCon AU 2017
• 31mBringing the Jewels of the Python World to Scala with SparkHolden Karau · Lambda World Seattle 2018
• 46mSpark: One Tool for Everything?Iskander Fakhrutdinov · Saint HighLoad++ 2025
• 36mNo more struggles with Apache Spark (PySpark) workloads in productionChetan Khatri · PyConZA 2019
• 50mBig Data Processing and Data Science Environment in PythonRick Bahague · Pycon PH 2017