PySpark - Data processing in Python on top of Apache Spark.

on YouTube ↗24m

" summary

PySpark exposes the Apache Spark programming model to Python and claims declarative DataFrame API queries get the same optimized speed as Scala via the catalyst optimizer while lower‑level Resilient Distributed Datasets stay lazy and fault‑tolerant. Demo loads JSON GitHub event files (tens of millions of rows), runs counts and top‑committer aggregations on a cluster (four machines ~40 cores each, ~1 TB total), and returns small results — live demo mostly succeeds though cluster UI visibility falters.

AI-written summary · about

" chapters (14)

1. 0:00Introduction
2. 2:55RDD
3. 4:41Transformations
4. 5:36MapReduce
5. 6:16Partitions
6. 7:17What is PySpark
7. 8:42How it works
8. 10:48Userdefined functions
9. 13:27Data Source
10. 14:05PySpark Data Format
11. 15:15Prediction Projection
12. 16:05DataFrame
13. 16:31Schema
14. 22:14Summary

" related talks

• 45mIndroduction to the PySpark DataFrame APIPeter Hoffmann · PyData Berlin 2015
• 50mBig Data Processing and Data Science Environment in PythonRick Bahague · Pycon PH 2017
• 01h 22mIntroduction to Big Data Processing using Spark and PythonRaoul-Gabriel Urma · PyData NYC 2018
• 43mPySpark in PracticeDat Tran, Ronert Obst · PyData London 2016
• 31mBringing the Jewels of the Python World to Scala with SparkHolden Karau · Lambda World Seattle 2018
• 25mScale Data Science by Pandas API on SparkXinrong Meng, Takuya Ueshin · PyData Global 2022
• 05mBig Data, No Big Deal: PySparkCatherine Devlin · PyOhio 2020
• 25mDistributed data workflows: PySpark vs DaskVaibhav Srivastav · DevConf.CZ 2020