An Introduction to Pandas 2, Polars, and DuckDB
summary
Shows concrete tradeoffs and outcomes: Pandas 2 optionally uses the PyArrow backend, yields ~50–70% string memory savings and copy‑on‑write behavior, and vectorized code beats rowwise apply by about 200x faster in the microbench shown. Polars runs on a Rust/Arrow engine with a lazy query optimizer that parallelizes groupbys (not distributed) and requires porting code because the API differs. DuckDB is an in‑process OLAP SQL engine that queries Pandas/Polars directly, reads CSVs in place, and often matches cluster performance on a single machine.
AI-written summary · about
chapters (16)
- 0:00Talk introduction
- 3:38Speaker background
- 6:51Why pandas 2 matters
- 9:21Pandas 2 features
- 13:15Chaining operations
- 18:26Vectorized vs apply
- 23:20PyArrow benefits
- 28:25Polars introduction
- 33:27Polars API and lazy eval
- 43:04Pandas vs Polars perf
- 47:22When to use Polars
- 53:58DuckDB intro
- 1:00:02DuckDB SQL queries
- 1:05:28Mixing Polars, pandas, and DuckDB
- 1:12:21Choosing between pandas, Polars, and DuckDB
- 1:18:03Q&A
related talks
34mData Wrangling with DuckDB
31mPolars on Spark
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
35mHow many dataframe libraries do you need to change a lightbulb?
32mThe PyArrow revolution in Pandas
40mDuckDB: Harnessing in-process analytics for data science and beyond
34mBridging Pandas and Polars
44mPolars; DataFrames in the multi-core era