Polars; DataFrames in the multi-core era
summary
Polars is an OLAP DataFrame engine written in Rust that targets modern hardware with cache‑aware data structures, lazy evaluation and a work‑stealing thread pool for effective parallelism. The optimizer removes duplicate computation (common‑subexpression elimination) and reports a 4.5× speedup on an example query. Polars Expressions compile to fast, vectorized operators so Python UDFs that take ~23 s become native expression pipelines, and a Rust plugins example turns a 23 s Python Pig Latin converter into ~2.5 s native code.
AI-written summary · about
chapters (11)
- 0:00Polars introduction
- 3:23Polars Expressions basics
- 7:15Why Expressions transform
- 10:53Performance benefits
- 13:24Query optimization
- 16:23Work-stealing parallelism
- 21:51Performance guarantees
- 23:40Polars plugins
- 27:56Pig Latin plugin
- 32:08Hybrid out-of-core
- 41:14Closing remarks
related talks
28mExpressive & fast dataframes in Python with polars
31mPolars on Spark
25mIntegrating Rust with Python...
01h 16mPower up your Polars code with Polars extention
37mPolars 1.0 and beyond
01h 22mAn Introduction to Pandas 2, Polars, and DuckDB
20mThe Polars vs SQL differences nobody is talking about
35mHow many dataframe libraries do you need to change a lightbulb?