Apache Parquet : the standard, efficient file format for tabular data
summary
Explains Parquet’s actual on-disk layout and tradeoffs: leaf columns become physical data pages <1MB grouped into column chunks and row groups, encodings (plain, RLE, dictionary encoding with 2 bits per index), then optional compression (Snappy, Zstd, lz4) and metadata like page index and Bloom filters with tunable size. Shows write-friendly S3 semantics — writes sequentially to S3 — and encryption per module, and notes ecosystem pieces (parquet-java, parquet-cpp/arrow, parquet-r) and performance caveats.
AI-written summary · about
chapters (16)
- 0:00Apache Parquet intro
- 1:23Why Parquet is open and efficient
- 2:06Row vs columnar storage
- 3:09Advantages of column storage
- 4:33Parquet format features
- 6:59Parquet file structure deep dive
- 10:15Handling optional and repeated values
- 13:48Encoding and compression in Parquet
- 18:06Data pages and column chunks
- 21:16Parquet file layout
- 22:44Optimizing queries with page index
- 25:13Bloom filters for filtering
- 27:40Parquet encryption
- 29:44Parquet ecosystem and tools
- 31:31Parquet limitations and future
- 36:30Q&A
related talks
29mEfficient and portable DataFrame storage with Apache Parquet
46mEfficient Data formats for Analytics with Parquet and Arrow
39mHow to avoid columnar calamities: what no one told you about Apache Parquet
08mBackstage interview
31mState of Parquet 2025: Structure, Optimizations, and Recent Innovations
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
39mORC and Parquet: Formats and Their Use on HDFS
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet