Breaking up arrays up into chunks for fun and science with Xarray and Dask
summary
Xarray plus Dask implement chunked, lazy n-dimensional processing of Landsat stacks: datasets are built from many scene files, registered in a Postgres-backed data cube and returned as an 54 GB lazy object that reads only requested chunks. Chunks default to sizes like 500x500x5 chunks, run on a threaded scheduler by default and scale to a distributed cluster; NDVI and time-mean reductions materialize by computing only the necessary chunk graph, and the system is deployed on NCI for real analyses.
AI-written summary · about
chapters (11)
- 0:00Xarray and Dask intro
- 3:41Xarray labeled arrays
- 9:12Dask task model
- 13:06Lazy eval and chunking
- 16:24Building Data Cube
- 18:10Computing NDVI
- 20:26RESTful API access
- 21:19Q&A chunk sizes
- 23:50Q&A sparse matrices
- 25:30Q&A MPI and distributed
- 26:36Q&A formats and backends
related talks
30mDask ❤️ Xarray: Geoscience at Massive Scale
30mGeoscience at Massive Scale
40mData-centric exploration using intake, dask, hvplot, and more
31mOpenDataCube - Petabyte Scale Satellite Imagery Processing in Python
01h 03mDask: Next Steps in Parallel Python
32mDemocratizing Distributed Computing with Dask and JupyterHub
44mUsing Dask for Parallel Computing in Python
01h 26mDask tutorial