Processing Large Geospatial Datasets with Dask & Xarray
summary
Demonstrates using Xarray and Dask Array to process ERA5 and other public datasets at hundreds of terabytes scale with concrete numbers like a 250 TB example split into 1.3 million chunks and benchmarks collected earlier in 2024. Shows a 30× array inflation from rolling-window constructs, explains why quantile computations and chunk fragmentation break clusters and how a rewritten quantile path fixes GIL contention, presents the first-ever geospatial benchmarks and an early query optimizer prototype that actually runs on 150 machines to finish 30 TB in minutes.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 1:08Dask and Xarray
- 6:26Geospatial benchmarks matter
- 12:09Bias correction deep dive
- 27:24Lessons learned
related talks
26mWho Said Wrangling Geospatial Data at Scale was Easy?
01h 24mData Science at Scale with Dask
03h 07mData of Unusual Size: Interactive Visualization
01h 03mDask: Next Steps in Parallel Python
29mBuilding Large Scale ETL Pipelines with Dask
01h 49mA practical guide to analysis & interactive visualization of massive datasets
01h 22mUsing Dask for Parallel Computing in Python
31mOpenDataCube - Petabyte Scale Satellite Imagery Processing in Python