Speed Up Data Access with PyArrow Apache Arrow Data is the new API

on YouTube ↗25m

" summary

Apache Arrow provides a language‑independent columnar memory format and PyArrow shows concrete wins: CSV read via pandas 1.7s versus PyArrow 129ms plus 65ms to convert (~200ms total), Parquet 600ms versus ~100ms, and a 126MB dataset transfers over Apache Flight in ~200ms (round trip ~400ms). It enables zero-copy reads for numeric non-null data, supports memory-mapped files to inspect 50GB without loading it all, and rehydrates buffers cheaply instead of full serialization.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 1:35What is Data
3. 6:32Data is the new API
4. 12:29What is Apache Arrow
5. 13:31Examples
6. 17:22Memory mapped files
7. 22:19Statistics

" related talks

• 33mArrow in Flight - New Developments in Data ConnectivityDavid Li · OSA Con 2022
• 32mThe PyArrow revolution in PandasReuven M. Lerner · PyCon US 2025
• 33mFree Your Esoteric Data Using Apache Arrow and PythonMaciej Wojton · PyData New York City 2019
• 32mConnecting PyData to other Big Data Landscapes using Arrow and ParquetUwe L. Korn · PyCon.DE 2017
• 29mFaster PySpark with Apache ArrowAllison Wang · PyData NYC 2024
• 35mPractical Applications of Apache ArrowMatthew Topol, Will Ayd · PyData Virginia 2025
• 22mAccelerating Analytics with Apache ArrowNeal Richardson · rstudio::conf 2019
• 34mLeveraging Apache Arrow for ML WorkflowsMatthew Topol · Øredev 2023