We don't dataframe shame: A love letter to dataframes
summary
Claims a formal data frame model and an algebra that reduces pandas’ ~600 functions to 15 operators, shows how parallelism splits into row-wise and column-wise classes, and compares real systems: pandas (legacy API), Spark (clustered, lazy, row-partitioned), Polars (Rust + Arrow) and Modin (distributed attempt). The group builds formal decomposition and partitioning rules, demonstrates feasibility of scaling dataframe semantics, and notes API compatibility and legacy behavior remain the hard practical obstacles.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:24Data analytics toolchain problem
- 3:31What is a dataframe
- 6:01Dataframe data model
- 9:10Dataframes vs matrices and DBs
- 11:16Dataframes as flexible food court
- 13:21Dataframe operations
- 15:48Parallelism in dataframes
- 17:33Pandas vs Apache Spark
- 20:43Polars and Rust dataframes
- 22:09Choosing the right dataframe
- 24:00Q&A
- 26:31Q&A: Unified APIs vs tools
- 27:35Q&A: Standardizing dataframe APIs
- 29:40Q&A: Distributing dataframe ops
- 32:35Q&A: Parallel Pandas support
- 34:01Q&A: Redesigning dataframes
- 35:23Q&A: Metadata enrichment
- 38:37Q&A: Dataframe performance
related talks
35mA Tour of the Many DataFrame Frameworks
30m"Better dataframes"
30mAn Introduction to DataFrames for pandas Users
18mLooking under the hood - demystifying data tools
22mDon’t Write Polars Code with a Pandas Accent
32mDataFrames: scaling up and out
36mScaling data workloads using the best of both worlds: pandas and Spark
24mPolyglot data with python: Introducing Pandas and Apache Arrow