Polars: A highly optimized dataframe library
summary
Polars 0.17 loads a 41k×83 CSV with scan_csv lazy + collect and through targeted casts and categoricals reduces memory from ~33 MB to ~2.5 MB (examples: float64→u16, ints→u8, strings→categorical). Regex extraction builds a speeds column, timezone-aware parsing converts created_on, nulls fill for EVs. Aggregations and pivot produce grouped means for plotting (then exported to pandas). Per-row apply crosses the Python boundary and runs ~100–300× slower, so prefer vectorized expressions.
AI-written summary · about
chapters (13)
- 0:00intro
- 0:22preface
- 1:00relevant background
- 1:38outline of opinions
- 2:32types
- 3:03ints
- 5:41strings
- 6:46extract ffs, speed & manual
- 9:16dates
- 10:24chain
- 12:30don't apply (if you can)
- 13:55master aggregation
- 18:34summary
related talks
31mBeyond Pandas: lightning fast in-memory dataframes with Polars
33mGet your Data prod ready, Fast, with Ruby Polars!
32mPandas, Polars and the DataFrame Consortium.
40mBeyond Pandas: The Great Python Dataframe Showdown
28mHow I used Polars to build built functime, a next gen ML forecasting library
31mPolars, the End of Spark?
28mIdiomatic Pandas
35mFast Pandas Performance Tricks You Wish You Knew Earlier