Apache Arrow: High-Performance Columnar Data Framework

on YouTube ↗01h 02m

" summary

Apache Arrow presents a stable 1.0 interprocess protocol (declared mid‑2020) and a component toolbox that yields near-zero-copy interoperability: the C data interface lets DuckDB, Velox and others exchange Arrow batches with no library dependency; Flight RPC layers Arrow on gRPC (with plans for libfabric/UCX data planes); Rust’s DataFusion provides a full query engine; C++ components implement hash joins, aggregations and near‑complete TPC‑H support. Parquet decoding is CPU heavy, so a faster Arrow‑native file format and lighter compression are proposed.

AI-written summary · about

" chapters (12)

1. 0:00Start
2. 2:27Recap
3. 9:26History of the Project
4. 12:32Building Relational Operators
5. 14:23Data Fusion
6. 28:12Augmented Columnar Layouts
7. 31:11Developing Alternative Data Transport Layers
8. 31:53Parallel Data Access
9. 33:56C Data Interface
10. 40:08Query Processing
11. 41:31Modular Reusable Relational Operators
12. 57:21The Story for Arrow and Gpus

" related talks

• 45mApache Arrow: In Pursuit of SpeedIgor Seliverstov · JPoint 2022
• 33mArrow in Flight - New Developments in Data ConnectivityDavid Li · OSA Con 2022
• 39mTaming the language border (...)Uwe L. Korn · Berlin Buzzwords 2019
• 46mEfficient Data formats for Analytics with Parquet and ArrowJulien Le Dem · Berlin Buzzwords 2016
• 34mApache Arrow as a full stack data engineering solutionAlessandro Molina · PyCon Italia 2023
• 32mVectorized Query Processing for CPUs using Apache ArrowJacques Nadeau · Scale By The Bay 2019
• 22mAccelerating Analytics with Apache ArrowNeal Richardson · rstudio::conf 2019
• 06mInterviewJacques Nadeau · Scale By The Bay 2019