Advanced Data Storage
summary
Hands-on demos use PyTables, netCDF4, numpy, pandas to create HDF5/NetCDF files, show a 100×100×100 array (1M elements, 4 bytes each) and compress it from ~8 MB down to 1.49 MB, and time writes (100 loops best ≈6 ms for an aligned chunk, misaligned chunks slow ~4×). Pandas HDF read is about ≈10× faster than CSV in the example. NetCDF4 supports time-unit conversions and a read-only multifile view that concatenates along the first unlimited (time) dimension to animate radar frames.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:42HDF5 format overview
- 9:52Tables in HDF5 with PyTables
- 12:28Metadata in HDF5 datasets
- 17:41Chunked arrays in HDF5
- 29:52Chunk size impact on performance
- 36:40Compression in HDF5 with PyTables
- 41:41HDF5 with pandas
- 49:07PyTables vs h5py
- 51:35netCDF4 format overview
- 1:02:59Time handling in netCDF4
- 1:10:30Multifile access with netCDF4
- 1:16:21Visualizing radar data with netCDF4
- 1:18:09Summary of HDF5 vs netCDF4
related talks
36mStore and manage data effortlessly with HDF5
45mInteractive Analysis of (Large) Financial Data Sets
44mUsing Dask for Parallel Computing in Python
34mSmall Big Data: using NumPy and Pandas when your data...
26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...
01h 16mIntro to Data Analysis with Python Data-table
28mWhy you should stop pretending your sparse data is dense
01mSmall Big Data: using NumPy and Pandas