Vectorized Query Processing for CPUs using Apache Arrow
summary
Explains a concrete execution path using Apache Arrow as the in-memory columnar format, JIT-compiling expression trees with Gandiva, and moving data with Arrow Flight; claims vectorized operators (hash join, aggregation) and CPU SIMD reduce instruction count and boost throughput — a 5M-row ODBC load drops from 54s (pyodbc) or 22s (turbo) to 1.3 seconds with pyarrow, and one customer workload shows up to 70x speedup; Dremio is implemented on these pieces and runs in production across hundreds of customers.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 4:55What is Apache Arrow
- 9:28Arrow components and structure
- 14:22Gandiva execution engine
- 20:17Columnar processing techniques
- 23:48Arrow Flight for data transport
- 26:37Future of Apache Arrow
- 27:27Q&A session
related talks
33mArrow in Flight - New Developments in Data Connectivity
01h 02mApache Arrow: High-Performance Columnar Data Framework
45mApache Arrow: In Pursuit of Speed
22mAccelerating Analytics with Apache Arrow
39mTaming the language border (...)
45mAccelerating OpenSearch With Streaming: Apache Arrow, Flight, Data...
28mArrow revolution in pandas and Dask
58mComposable Data Systems, the new frontier