Polyglot data with python: Introducing Pandas and Apache Arrow
summary
Claims that Apache Arrow + pandas let Python exchange data with other ecosystems without expensive serialization, demonstrates reading a Spark-created Parquet file into pandas via pyarrow, manipulating schema and metadata, then writing back. Uses a ~71k rows sample and shows Parquet ↔ Arrow ↔ pandas roundtrips (Arrow memory mapping avoids full deserialization). Notes disk size tradeoffs and repeats that no CSV at scale — practical demos and working code are provided.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:04About me
- 1:40The problem
- 4:32CSV doesnt scale
- 5:09Data modification
- 14:00Protocols
- 19:29Pioneer rule
- 20:04Demo
- 21:07Pandas Dataframe
- 22:03Pandas Metadata
- 22:26Pandas File System
- 23:01Python
- 23:29Documentation
- 23:49Outro
related talks
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
31mThe Arrow revolution in pandas and Dask
28mHigh speed data from the Lakehouse to DataFrames with Apache Arrow
29mFaster PySpark with Apache Arrow
28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.
46mInternals of Speeding up PySpark with Arrow