Geoscience at Massive Scale
summary
A concrete account of scaling geoscience: a 250 TB catalog is sampled to a 12 TB subset for ~$2 of cloud compute, the cluster holds ~3.5 TB in memory while the task graph explodes to ~70,000 tasks for naive rechunking, and optimized work reduces that to ~15,000 tasks. Key changes are visible: Dask + Xarray power the pipeline, scheduler and memory fixes plus a P2P rechunking path shift rechunking from RAM to disk, change task growth from superlinear to linear, and make large-array summaries far more stable.
AI-written summary · about
chapters (10)
- 0:00Geoscience at massive scale
- 1:51Processing 12TB geospatial data
- 5:48Dask and xarray for distributed geoscience
- 10:23Geoscience computation challenges
- 11:07Improving array API with P2P rechunking
- 17:35Downscaling and task explosion
- 20:24Need for geospatial benchmarks
- 23:15Building geospatial benchmarks
- 23:58Conclusion and future
- 24:20Q&A on P2P rechunking
related talks
30mDask ❤️ Xarray: Geoscience at Massive Scale
38mScale out from the very beginning
01h 17mScaling PyData with Dask
28mBreaking up arrays up into chunks for fun and science with Xarray and Dask
38mBig Data Oceanography
34mGeoscience Big Data Computing with Python
40mScaling Python - Bank Edition
01h 03mDask: Next Steps in Parallel Python