High speed data from the Lakehouse to DataFrames with Apache Arrow
summary
Claims use Apache Arrow end-to-end with Pandas2, Polars and DuckDB for zero-copy transfers and replace JDBC/ODBC with Arrow Flight (and ADBC) protocols. A query service is built using DuckDB to run point-in-time correct joins on Parquet-backed Lakehouse tables and deliver Arrow tables to Python clients. Results show interactive performance—reads of roughly 2–3 million rows/sec and “millions of rows in seconds” for training-data creation, so the service works as a low-latency path from lake to DataFrames.
AI-written summary · about
chapters (9)
- 0:00Lakehouse data access with Arrow
- 2:15Python access to Lakehouse data
- 7:30Arrow solves transfer bottlenecks
- 11:19DuckDB for Lakehouse access
- 13:48Temporal joins for training data
- 19:50SQL complexity with feature stores
- 21:58Performance benchmarks
- 24:47Live demo: interactive access
- 26:11Conclusion & Q&A
related talks
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
24mPolyglot data with python: Introducing Pandas and Apache Arrow
09mBackstage interview
42mColumnar Storage: Redefining Data Management for the Modern Era
31mThe Arrow revolution in pandas and Dask
46mInternals of Speeding up PySpark with Arrow
33mYou Don’t Need Spark for That: Pythonic Data Lakehouse Workflows
27mApache Arrow - The columnar format! What's that?