All Them Data Engines: Data Munging with Python circa 2023
summary
Compares concrete tools (Pandas, NumPy, Spark, Dask, Polars, Arrow) on the MovieLens data and shows runnable notebooks and Colab links in a GitHub repo; demonstrates CSV quirks (commas inside titles), builds a pure‑Python group‑by and a parser, measures performance (example: NumPy 22× faster for a 10M sum, ~50% memory reduction versus lists), reports counts (9,742 movies), contrasts eager vs lazy execution and shows Spark’s UI; notebooks run end‑to‑end with minor setup hiccups.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 10:49Python data structures
- 27:41MovieLens analysis
- 1:06:40Python collections
- 1:13:10Numpy for faster data processing
- 1:29:13Pandas wrangling
- 1:40:24Spark basics
- 1:51:13Data engines comparison
related talks
31mThe PyData map: Presenting a map of the landscape of PyData tools
34mDataComPy - Dataframe Comparisons made Explicit
25mPandas on Jupyter: Improving Your Data Analysis Workflow
41mBetter Code for Data Science
46mThe Python Data Ecosystem: Navigating a fragmented landscape
33mYou Don’t Need Spark for That: Pythonic Data Lakehouse Workflows
42mSimplifying Data Analysis with GitHub Codespaces, Jupyter Notebooks & Open AI
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python