Efficient and portable DataFrame storage with Apache Parquet
summary
Apache Parquet stores column chunks split into ~16 KB pages, uses dictionary encoding plus run‑length and bit‑packing before codecs like snappy, and supplies min/max stats at page/column/row‑group levels so engines skip irrelevant data. A 1.6 GB CSV example compresses to about a fifth of its size; reads scale across cores and Python bindings via pyarrow and fastparquet work, writes are possible but non‑appendable and typically ~1.5× slower than reads.
AI-written summary · about
chapters (9)
- 0:00Apache Parquet introduction
- 1:02Parquet format history
- 3:53Efficient data storage
- 8:10Encoding techniques
- 13:58Compression algorithms
- 15:45Query pushdown benefits
- 19:51Portability and support
- 21:58Arrow with Parquet
- 23:27Q&A Parquet vs HDF5
related talks
39mHow to avoid columnar calamities: what no one told you about Apache Parquet
37mApache Parquet : the standard, efficient file format for tabular data
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
46mEfficient Data formats for Analytics with Parquet and Arrow
08mBackstage interview
20mGive me the Data
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
42mColumnar Storage: Redefining Data Management for the Modern Era