The PyArrow revolution in Pandas
summary
Pandas experiments with PyArrow show concrete wins and clear trade‑offs: CSV reads drop from ~103s to ~8.6s (≈10×), binary formats shrink a 2.2GB CSV to parquet 379MB (feather 1.4GB), memory can fall from ~15.6GB to ~3.7GB, and operations like regex searches and self‑joins run several× faster (e.g., 2s→366ms, 11s→102ms). Row‑oriented indexing and some per‑row operations remain much slower; PyArrow dtypes are nullable and interoperable and pandas plans PyArrow integration going forward.
AI-written summary · about
chapters (13)
- 0:00PyArrow and Pandas intro
- 1:26DRY in Pandas foundation
- 3:32Pandas memory inefficiency
- 5:39Apache Arrow and PyArrow
- 7:26Faster CSV read/write
- 10:35Faster file formats with Arrow
- 13:01Swap NumPy for PyArrow
- 22:48PyArrow nullable types
- 25:15PyArrow string handling
- 26:23Pandas swappable backends
- 27:31Q&A performance impact
- 28:37Q&A Pandas in data ecosystem
- 30:22Q&A NumPy vs PyArrow
related talks
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
01h 22mAn Introduction to Pandas 2, Polars, and DuckDB
40mBeyond Pandas: The Great Python Dataframe Showdown
28mArrow revolution in pandas and Dask
29mFaster PySpark with Apache Arrow
33mFree Your Esoteric Data Using Apache Arrow and Python
41mFaster SQL with pandas and Apache Arrow
34mBridging Pandas and Polars