Indroduction to the PySpark DataFrame API

on YouTube ↗45m

" summary

PySpark demonstrates the Spark DataFrame API (introduced in Spark 1.3) as a declarative layer that compiles queries to JVM bytecode so Python data frames get same execution speed as Scala. The demo loads JSON and CSV, reads Parquet, joins employees and salaries, caches results, registers user-defined functions distributed to workers, and pulls a reduced result into pandas. A toy dataset returns 1797 male and 120000 female and highlights that RDDs in Python suffer extra serialization overhead.

AI-written summary · about

" chapters (17)

1. 0:00Introduction
2. 1:14Apache Spark core concepts
3. 7:44PySpark basics and cluster flow
4. 9:55First RDD demo
5. 14:31PySpark DataFrame API
6. 15:14Spark SQL and DataFrame overview
7. 17:43DataFrame API performance
8. 18:47Creating DataFrames from sources
9. 20:16Catalyst optimizer and execution plans
10. 25:26First DataFrame tweet example
11. 28:13Joining CSV datasets
12. 31:19Caching DataFrames
13. 34:18Grouping and aggregating data
14. 36:06User-defined functions
15. 36:50PySpark with pandas
16. 37:54Summary of benefits
17. 40:23Q&A

" related talks

• 24mPySpark - Data processing in Python on top of Apache Spark.Peter Hoffmann · EuroPython 2015
• 01h 22mIntroduction to Big Data Processing using Spark and PythonRaoul-Gabriel Urma · PyData NYC 2018
• 50mBig Data Processing and Data Science Environment in PythonRick Bahague · Pycon PH 2017
• 30mImproving PySpark Performance: Spark performance beyond the JVMHolden Karau · PyCon AU 2017
• 25mScale Data Science by Pandas API on SparkXinrong Meng, Takuya Ueshin · PyData Global 2022
• 01h 13mFrom Pandas to PySparkCynthia Ukawu · PyData Virginia 2025
• 36mScaling data workloads using the best of both worlds: pandas and SparkEng & Kwon · PyData Seattle 2023
• 41mImproving PySpark Performance Spark performance beyond the JVMHolden Karau · PyData DC 2016