Small Big Data: using NumPy and Pandas when your data doesn't fit ...
summary
Explains concrete tactics to handle data too big for RAM using NumPy and Pandas: lossless dtype downcasting (64→16-bit cuts memory ~4x), sparse arrays for mostly-zero data, on-disk chunking (process 1k rows at a time to compute global max), Zarr/HDF-like chunked storage, and indexing via SQLite to fetch only matching rows—example shows SQLite index gives 50x speedup versus scanning CSVs; includes when renting more RAM is the simpler choice.
AI-written summary · about
chapters (17)
- 0:00Small Big Data
- 0:14Prelude: the most important question
- 1:11TIME FOR A BIG DATA CLUSTER!!!!
- 2:13A non-solution: don't use RAM, just disk
- 4:05The software solution: use less RAM
- 7:18Compression: Numpy dtypes
- 8:52Compression: sparse arrays
- 11:14Compression: Pandas dtypes When loading data you can specify types
- 13:54Chunking: loading Numpy chunks with Zarr
- 15:22Chunking: with Pandas
- 18:52Indexing: the simplest solution
- 19:26Indexing: Pandas without indexing
- 20:45Indexing: populate SQLite from Pandas
- 22:10Indexing: load from SQLite into DataFrame
- 23:01Indexing: SQLite vs. CSV
- 24:14Conclusion: what about other libraries?
- 25:06Conclusion: don't forget about
related talks
34mSmall Big Data: using NumPy and Pandas when your data...
01mSmall Big Data: using NumPy and Pandas
31mStrategies for Working with Data as It Grows
32mDataFrames: scaling up and out
14mKeynote: Scalable Python We Can Afford
35mData Processing Pipelines for Small Big Data
01h 20mAdvanced Data Storage
40mBeyond Pandas: The Great Python Dataframe Showdown