Accelerating Analytics with Apache Arrow
summary
The Arrow R package binds to the Arrow C++ library and adds a data sets API so a directory of 125 Parquet files (37 GB) becomes a single table with partition keys exposed as virtual columns year and month; open_dataset plus dplyr scans 2 billion rows in under 4 seconds while pushing work into file chunks, Parquet yields order-of-magnitude smaller files, and Spark integration gives 40x speed up pulling data via sparklyr.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:31What is Apache Arrow
- 9:12Why use Apache Arrow
- 14:27Try the package
- 14:39Conda recipe
- 14:55Linux
- 15:23Builds
- 16:18Community Support
- 16:55URSA Labs
- 18:08Questions
related talks
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
04mAccelerating geospatial computing using Apache Arrow
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
46mEfficient Data formats for Analytics with Parquet and Arrow
33mFree Your Esoteric Data Using Apache Arrow and Python
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
01h 02mApache Arrow: High-Performance Columnar Data Framework
33mArrow in Flight - New Developments in Data Connectivity