A Tour of the Many DataFrame Frameworks
summary
Compares pandas, Modin, Dask and Polars with concrete internals: API compatibility and migration gaps, partitioning strategies, memory layouts (NumPy arrays vs Arrow columnar format), execution models (single‑threaded eager vs lazy optimized graphs), and Polars’ Rust copy-on-write design. Shows short code samples, points out Modin/Dask fallbacks that destroy performance, explains when a framework runs out of features, and gives practical tradeoffs for rewriting versus drop-in replacement.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:26What is a data frame
- 7:43Pandas API complexity
- 13:07Modin vs Dask vs DuckDB
- 15:55Polars API design
- 16:17Memory management
- 18:45Partitioning and parallel
- 21:32Execution models
- 25:51Query optimization
- 26:53Python data frames
- 27:58Under the hood
- 28:40Q&A
related talks
30m"Better dataframes"
41mWe don't dataframe shame: A love letter to dataframes
35mHow many dataframe libraries do you need to change a lightbulb?
01h 32mAn Introduction to Polars
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
22mDon’t Write Polars Code with a Pandas Accent
30mSpeed is Not All You Need for Data Processing
39mDesigning for high-performance execution with Arrow and Polars