Building NumPy Arrays from CSV Files, Faster than Pandas
summary
Claims a C extension delimited_to_arrays() that reuses Python’s csv.reader tokenizer, collects Unicode code point lines per column, then converts directly to C types and writes into NumPy arrays with minimal PyObject work. Supports all NumPy dtypes, optional per-column type discovery, and benchmarks on 10M rows showing performance often better than pandas read_csv (order‑of‑magnitude faster on wide tables, slightly slower for uniform float columns). StaticFrame integrates it for DataFrame construction.
AI-written summary · about
chapters (16)
- 0:00Start
- 2:09csv.reader(): Configurable Dialect Configuration
- 2:22csv.reader(): Not Implemented
- 2:51Pandas: read_csv()
- 5:12Diagrams & Flowcharts
- 6:32Challenges of Building Column Arrays from CSV
- 7:56Avoiding PyObjects
- 10:31Interface of delimited_to_arrays()
- 12:14Usage in StaticFrame
- 13:29C Structs: DelimitedReader
- 14:08C Structs: CodePointLine (CPL)
- 19:24Benchmarks
- 20:22Nine DataFrame Fixtures
- 22:37Performance Observations
- 24:14Parallelization
- 24:55Parallelized CSV Loading
related talks
28mEmploying NumPy's NPY Format for Faster Than Parquet DataFrame...
31mFaster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
32mYou don't always need NumPy
39mDyND: Enabling complex analytics across the language barrier
50mNumPy Versus APL
33mExtending Pandas using Apache Arrow and Numba
01h 20mAdvanced Data Storage
36mStaticFrame: An Immutable Alternative to Pandas