Beyond Pandas: The Great Python Dataframe Showdown
summary
Compares pandas limits to concrete alternatives: Arrow shows memory-mapped CSV→Parquet conversion that drops a 1.9 GB CSV to 1.9 GB to <0.5 GB on disk; Vex executes expression-based lazy queries over 61 million rows without loading everything into RAM; Polars composes lazy expressions and evaluates optimized query plans in Rust for very fast in-memory aggregations. Live demos run end-to-end and the conversions and queries complete successfully.
AI-written summary · about
chapters (16)
- 0:00Start
- 2:51Pandas
- 5:31Alternatives to Pandas
- 6:39Memory Format
- 7:59Vex
- 8:59Polars
- 11:38Demo
- 12:10How To Load Data with Pyro
- 12:48Csv Reading
- 13:38Chunked Array
- 14:27Create a Memory Map
- 17:15Vex Example
- 18:40Expressions
- 23:39Polish Expressions
- 24:58Expressions To Work with Lists of Values
- 26:41Query Plan
related talks
32mThe PyArrow revolution in Pandas
31mFaster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
30mIs the great dataframe showdown finally over? Enter: polars
28mExpressive & fast dataframes in Python with polars
38mExpressive and fast dataframes in Python with polars
01h 17mPandas from the Inside / "Big Pandas"
30m"Better dataframes"
33mSkinny Pandas Riding On A Rocket