Efficient Data formats for Analytics with Parquet and Arrow
summary
Compares Parquet (disk-optimized) and Arrow (memory-optimized) with concrete tradeoffs: Parquet uses columnar storage, statistics and encodings like dictionary and bit-packing to cut I/O and skip blocks; Arrow defines a cross-language in‑memory layout of record batches with contiguous vectors, SIMD‑friendly access and zero-copy IPC/RPC so systems send buffers without serialization. Provides C++/Java bindings, integrates with Spark/Impala/Drill/Kudu/Pandas, and shows immutable batches, spill-to-disk strategies and working prototypes that already remove serialization overhead.
AI-written summary · about
chapters (12)
- 0:00Introduction and overview
- 1:00Benefits of columnar layout
- 4:29Parquet vs. Arrow
- 6:49Parquet internals
- 9:12Arrow and CPU performance
- 14:10Cache locality and SIMD
- 17:35Community and standards
- 22:14Ecosystem integration
- 25:59Data format details
- 30:03Zero-copy IPC and RPC
- 34:28Future steps and conclusions
- 36:44Q&A session
related talks
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
01h 02mApache Arrow: High-Performance Columnar Data Framework
22mAccelerating Analytics with Apache Arrow
37mApache Parquet : the standard, efficient file format for tabular data
33mFree Your Esoteric Data Using Apache Arrow and Python
42mColumnar Storage: Redefining Data Management for the Modern Era
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API