How many dataframe libraries do you need to change a lightbulb?
summary
Compares pandas, Polars, and DuckDB by concrete limits and wins: pandas fails on a 50 GB dataset on a single laptop while Polars and DuckDB stream and complete joins in about 2 minutes; Polars supports both eager and lazy execution and ships a Rust-backed expression engine, DuckDB is an analytics-first SQL engine (only lazy) that integrates well with LLMs, and real Rust Rust plugin extensions (geospatial/address lookup) are built and run blazing fast.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:51Pandas origin and dominance
- 3:39Pandas strengths and limits
- 7:11Introducing Polars
- 14:42DuckDB for analytics
- 20:30Extensibility and plugins
- 25:57Syntax differences
- 30:14Future roadmap
- 33:28Narwhals for maintainers
- 33:50Conclusion
related talks
01h 22mAn Introduction to Pandas 2, Polars, and DuckDB
30m"Better dataframes"
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
30mSpeed is Not All You Need for Data Processing
34mData Wrangling with DuckDB
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
37mSpark, Dask, DuckDB, and Polars: Benchmarks
28mAnalyze data easily with duckdb - and the implications on data architectures