Introduction to Big Data Processing using Spark and Python

on YouTube ↗01h 22m

" summary

Hands-on workshop that walks from Hadoop concepts to running PySpark code: it explains batch vs streaming, shows RDD and DataFrame patterns (map, flatMap, filter, reduce, groupByKey), demonstrates a Google Colab notebook preconfigured with examples, warns that Python API incurs JVM serialization overhead, and spins up an AWS EMR cluster reading ~126 million JSON records from S3 (count finishes in minutes) so the code is built and actually runs at cluster scale.

AI-written summary · about

" chapters (9)

1. 0:00Welcome
2. 5:04Big data challenges
3. 22:23Hadoop ecosystem
4. 36:15Apache Spark intro
5. 43:19Spark RDD basics
6. 57:04Processing War and Peace
7. 1:08:46Scaling Spark on AWS EMR
8. 1:17:20Live distributed demo
9. 1:21:35Next steps

" related talks

• 50mBig Data Processing and Data Science Environment in PythonRick Bahague · Pycon PH 2017
• 44mUsing Python with Hadoop to create production ready big data applicationsUlrich Zink · PyData London 2016
• 24mPySpark - Data processing in Python on top of Apache Spark.Peter Hoffmann · EuroPython 2015
• 43mPySpark in PracticeDat Tran, Ronert Obst · PyData London 2016
• 41mKeynote: Making the Big Data ecosystem work together with PythonHolden Karau · PyData London 2018
• 01h 13mFrom Pandas to PySparkCynthia Ukawu · PyData Virginia 2025
• 04h 02mHands-on Session on Data Science for BeginnersAishwarya Verma, Krishna Rastogi · SkillUp 2021
• 39mHow Python Drives Modern Data EngineeringKiana Alessandra Villaera · PyCon Philippines 2023