Pandas 2, Dask or Polars? Tackling larger data on a single machine
summary
Pandas 2 adds Arrow types and copy‑on‑write (to be default later) and on one dataset shows 82 million rows → 39 GB (numpy) vs 11 GB (Arrow) but mixed perf where some groupbys get slower. Polars is Rust + Arrow, multicore and often much faster — e.g. Polars: 3s vs Pandas 30s on a median/sort aggregate — and its lazy scan handles many parquet files in 13s for 640M rows with optional streaming mode. Dask with new Expressions cuts a naive 402s run to 26s by query planning.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:06Choosing pandas, Dask or Polars
- 3:53Pandas 2 Arrow backend
- 7:24Copy-on-write in pandas
- 10:35Polars architecture
- 13:01Benchmarking pandas, Polars and Dask
- 15:49Lazy evaluation in Polars
- 17:33Scaling with Dask
- 20:22Polars vs Dask for medium data
- 22:47Key differences and migration
- 24:53Summary
- 25:57Q&A
related talks
01h 17mPandas from the Inside / "Big Pandas"
01h 02mPandas from the inside
29mHow Narwhals is silently bringing pandas, Polars, DuckDB, PyArrow, and more together
31mThe PyData map: Presenting a map of the landscape of PyData tools
34mDataComPy - Dataframe Comparisons made Explicit
31mPolars Plugins: how you (yes, you!) can extend Polars Dataframes
33mYou Don’t Need Spark for That: Pythonic Data Lakehouse Workflows
43mThe Column's the limit