Free Your Esoteric Data Using Apache Arrow and Python
summary
Shows concrete pipelines and numbers: reading CSV with Apache Arrow drops measured latency from 23 ms → 8 ms and JSON from 144 ms → 15 ms, converting strings to categories cuts memory from 10 MB → 1 MB, and writing Parquet shrinks disk size by ~4x. Provides code that parses C++ headers with clang + template generation to produce Arrow/Parquet converters, produces one Parquet file per message type, and reports that the generated converters run fast enough for CI and real analysis workflows.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 0:44Esoteric data importance
- 2:29Challenges with esoteric data
- 4:36Pandas for data analysis
- 6:21Unaligned dataset alignment
- 8:06Frequency analysis for unalignable data
- 9:32Freeing data with Arrow
- 10:13Why PyArrow
- 12:43PyArrow speed and memory
- 13:47Strings to categories
- 15:12Parsing arbitrary formats
- 16:57Parquet storage and compression
- 17:19Hierarchical data in Arrow
- 18:24C++ to Arrow conversion
- 21:57Schema from C++ with Clang
- 24:03C++ converters with Jinja2
- 25:27Storing in Parquet files
- 26:34Automating generated code
- 27:15Arrow vs pandas?
- 28:39Arrow cross-language integration
- 29:25Sharing Arrow tables
- 30:29Arrow production stability
- 31:12Real-world Arrow benefits
related talks
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
22mAccelerating Analytics with Apache Arrow
46mEfficient Data formats for Analytics with Parquet and Arrow
35mPractical Applications of Apache Arrow
34mLeveraging Apache Arrow for ML Workflows
33mArrow in Flight - New Developments in Data Connectivity
42mState of the Apache Arrow Ecosystem:How your project can leverage Arrow!