An Introduction to PySpark
summary
Compares pandas and PySpark on Brisbane library checkout logs, shows code that groups checkouts and uses window functions to group checkouts within five seconds and finds a burst of checkouts spanning about 68 seconds, demonstrates that PySpark is a thin Python API over a Scala/Java engine so collect triggers Java pickling, highlights a 2021 benchmark where Spark only beats pandas on large datasets, explains lazy transformations versus eager actions causing repeated work and plan-size out‑of‑memory errors, and mentions Apache Sedona for geospatial needs.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:25What is PySpark
- 3:12PySpark vs pandas
- 5:18Grouping checkouts by time
- 7:02Benchmarking PySpark
- 8:47PySpark's Java underpinnings
- 10:12Why distributed processing
- 11:59MapReduce and Spark
- 14:23Out of memory errors
- 15:46Lazy vs eager operations
- 17:49Java stack traces
- 19:14Non-deterministic joins
- 20:18When to choose PySpark
- 21:46Conclusion
related talks
38mPySpark in Practice
36mScaling data workloads using the best of both worlds: pandas and Spark
18mLooking under the hood - demystifying data tools
44mProcessing large data with Apache Spark
26mLarge Scale Data Validation (with Spark and Dask)
46mSparking Success: Unveiling the Journey of Apache Spark Application Development
20mIntroduction to Zeppelin Notebooks and PySpark 2 0
43mIntroduction to Spark Datasets