Employing NumPy's NPY Format for Faster Than Parquet DataFrame...
summary
Claims that NPY/NPZ with JSON metadata and StaticFrame outperform Parquet with concrete numbers: writing a 10k×10k float frame drops from about 10.5s to 1.42s and reading from about 5s to 1s, an order‑of‑magnitude write speedup and large read gains. Implementation reimplements NPY authoring, rejects object arrays, caches headers, and exports NPY files to directories for memory‑mapping full DataFrames. Benchmarks scale to 100M elements where NPZ often beats uncompressed Parquet and memory‑mapped NPYs can even outpace pickle.
AI-written summary · about
chapters (24)
- 0:00Intro
- 1:16The Quest for Complete DataFrame Serialization
- 1:48NumPy Enhancement Proposal (NEP) 1
- 3:13Promising Performance of NPZ versus Parquet
- 4:12Overview
- 4:49Components of a DataFrame
- 7:54Block-Consolidation Strategies Unconsolidated Blocks
- 8:59Block Consolidation & Complexity
- 11:33The NPY Format
- 12:01Converting Contiguous Bytes to an Array
- 13:48NPY & Object Arrays
- 14:15NPY Versions
- 14:45The NPZ Format
- 15:18Encoding a DataFrame as an NPZ
- 16:21JSON Metadata
- 17:25NPY Performance in Numpy
- 18:35Lies, Damned Lles, and Benchmarks
- 19:34Nine DataFrame Fixtures
- 23:37Memory Maps
- 23:54Memory Mapping an Array
- 25:29Memory Mapping a DataFrame
- 26:18Current State
- 27:24Future Work
- 27:44Conclusions
related talks
29mEfficient and portable DataFrame storage with Apache Parquet
31mFaster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
40mBeyond Pandas: The Great Python Dataframe Showdown
39mHow to avoid columnar calamities: what no one told you about Apache Parquet
36mStaticFrame: An Immutable Alternative to Pandas
30mParticle physics, 10,000 times faster
34mSmall Big Data: using NumPy and Pandas when your data...
01h 02mPandas from the inside