Beyond Pandas: lightning fast in-memory dataframes with Polars
summary
Benchmarks claim Polars runs typical groupby and join workloads 13-16x faster than Pandas (DuckDB/DB‑Benchmark), using a Rust + Arrow engine; lazy querying via scan_csv/lazy collect enables predicate and projection pushdown and streaming larger‑than‑RAM pipelines so a 20 GB CSV (≈90M rows) produces results on a 32 GB machine that Pandas cannot. Integration uses Arrow for zero-copy with Pandas 2.0, and approximate cardinality and sinks to Parquet handle huge outputs.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker intro
- 3:12Pandas and Spark limitations
- 7:46Alternatives to Pandas overview
- 11:57Polars introduction and growth
- 13:40Polars performance features
- 16:29Lazy evaluation in Polars
- 19:19Using Polars in production
- 20:22Personal benchmark results
- 23:28Polars current limitations
- 24:54Why switch to Polars
- 25:35Q&A session
