Practical Applications of Apache Arrow
summary
Apache Arrow specifies a columnar in‑memory layout that is identical on the wire so systems achieve zero-copy IPC across C, Python, Go, Java and more and avoid serialize/deserialize costs; real deployments show big wins: DuckDB via ADBC removes transpose overhead versus ODBC/JDBC, Snowflake shrinks a client from 50MB→5MB using nanoarrow, and a file‑writer (pantab) sees ~30x throughput improvement — these tools interoperate and the implementations measurably work.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:50Apache Arrow overview
- 4:20Arrow memory layout
- 8:13Cross-language data sharing
- 10:42ADBC for databases
- 14:36Arrow in Polars/Pandas
- 18:11Arrow interoperability
- 22:46Future of composable data systems
- 25:57Q&A
related talks
33mArrow in Flight - New Developments in Data Connectivity
42mState of the Apache Arrow Ecosystem:How your project can leverage Arrow!
32mThe expanding Apache Arrow universe
27mApache Arrow - The columnar format! What's that?
01h 04mWhere We’re Going, We Don’t Need Rows: Columnar Data Connectivity with Apache Arrow ADBC
34mLeveraging Apache Arrow for ML Workflows
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
36mEmbrace the chaos: Composable data systems with fewer asterisks