Dask ❤️ Xarray: Geoscience at Massive Scale
summary
Demo loads a 250 terabyte National Water Model and subsets to 18 terabytes then runs an Xarray-on-Dask pipeline across a Coiled cluster of about 100 VMs, producing a 210 MB result for roughly $3. Benchmarks reveal hotspots: task explosion (110,000+ tasks, legacy rechunking created hundreds of thousands of tiny chunks), fixed by making P2P rechunking default and reworking downscaling so chunk counts drop from ~300k to ~40k and memory and runtime scale linearly.
AI-written summary · about
chapters (12)
- 0:00Dask and Xarray intro
- 2:5118TB geospatial processing
- 5:02Monitor Dask cluster
- 7:10Dask for parallel Python
- 9:18Xarray labels for NumPy
- 11:27Scaling Xarray with Dask
- 12:11Dask Xarray API improvements
- 14:42P2P re-chunking cuts overhead
- 19:03Downscaling with rolling means
- 22:14Geospatial benchmarks
- 24:43Key takeaways and future
- 25:50Q&A
related talks
30mGeoscience at Massive Scale
28mBreaking up arrays up into chunks for fun and science with Xarray and Dask
01h 03mDask: Next Steps in Parallel Python
01h 23mData Science at Scale with Dask
31mDask at Global Scale with
01h 26mDask tutorial
32mDemocratizing Distributed Computing with Dask and JupyterHub
01h 17mScaling PyData with Dask