Apache Arrow as a full stack data engineering solution
summary
Apache Arrow is presented as the de facto standard for data interchange in big data and data engineering, and the talk explains how PyArrow provides more than just a file format. It describes using PyArrow to build a complete stack — including storage, in-memory analysis (interoperable with tools like pandas), and network transmission — to implement end-to-end data engineering solutions.
AI-written summary · about
chapters (5)
- 0:00Apache Arrow intro
- 13:56Arrow full stack solution
- 21:40Use cases and demos
- 27:12Performance vs ecosystem
- 32:31Q&A
related talks
45mApache Arrow: In Pursuit of Speed
01h 02mApache Arrow: High-Performance Columnar Data Framework
42mState of the Apache Arrow Ecosystem:How your project can leverage Arrow!
33mArrow in Flight - New Developments in Data Connectivity
27mApache Arrow - The columnar format! What's that?
35mPractical Applications of Apache Arrow
32mThe expanding Apache Arrow universe
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet