Data Oriented Programming
summary
Claim: CPU cycles are cheap but memory access is the bottleneck, so optimizing data layout beats naive parallelism. Examples and numbers: numpy 1.6s -> multiprocessing 1.2s, numexpr 0.14s -> 0.11s by expression rewriting and blocking to avoid large temporaries, giving ~3× speedups over memory‑inefficient C paths. Tools and results: columnar Bcolz plus blosc compression shrinks a 1.5GB pandas dataset to <100MB and yields faster queries, with blosc reported up to 5× faster than memcpy for streaming decompression.
AI-written summary · about
chapters (10)
- 0:00Data oriented programming intro
- 1:14CPU speed no longer bottleneck
- 10:57Starving CPU problem
- 14:53CPU cache hierarchy
- 17:23Blocking for cache efficiency
- 18:28Numex vs NumPy performance
- 20:40Optimal data containers for big data
- 24:18Blosc and columnar storage
- 30:13MovieLens benchmarks
- 33:04Summary
related talks
32mDataFrames: scaling up and out
22mDon’t Write Polars Code with a Pandas Accent
30mSpeed is Not All You Need for Data Processing
30mParticle physics, 10,000 times faster
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
39mDesigning for high-performance execution with Arrow and Polars
31mStrategies for Working with Data as It Grows
01h 21mPandas from the Inside