Taming the language border (...)
summary
Apache Arrow provides a language‑neutral columnar in‑memory format and libraries that let Java produce off‑heap Arrow record batches, hand the memory pointers to Python or R without serializing, and serve data over Flight/GPRC to avoid intermediate copies. Production pieces include vectorized Parquet readers, an LLVM microkernel (Gandiva) and zero-copy cross-language pointers, with six language implementations and reported 2–40× speedup for Spark UDFs when avoiding row-wise serialization.
AI-written summary · about
chapters (5)
- 0:00Start
- 13:05Vectorized Packet Readers
- 27:05Questions
- 27:16Questions Round
- 32:40Encode the Schema in the Memory
related talks
01h 02mApache Arrow: High-Performance Columnar Data Framework
33mArrow in Flight - New Developments in Data Connectivity
34mLeveraging Apache Arrow for ML Workflows
45mApache Arrow: In Pursuit of Speed
42mState of the Apache Arrow Ecosystem:How your project can leverage Arrow!
09mBackstage interview
46mEfficient Data formats for Analytics with Parquet and Arrow
32mVectorized Query Processing for CPUs using Apache Arrow