Democratizing Distributed Computing with Dask and JupyterHub
summary
A cloud deployment combines JupyterHub, JupyterLab, XArray and Dask on Kubernetes to let scientists run real analyses: a 70 GB sea‑level array is split across workers, an averaging job runs in ~30 seconds, clusters scale to thousand‑core clusters on demand, and the public Pangea site provides accounts to reproduce the demo. The claim is reproducible operational infrastructure, not theory, and it demonstrably reduces interactive turnaround for large raster datasets using Dask+XArray.
AI-written summary · about
chapters (7)
- 0:00Start
- 0:05Matthew Rocklin
- 12:53Dashboard
- 15:09Plotting the Results
- 16:56Jupiter Notebook
- 17:35Jupiter Hub
- 24:37Issue Tracker
related talks
39mHow We Hosted 200+ Data Scientists in a K8S Cluster
01h 17mScaling PyData with Dask
01h 23mData Science at Scale with Dask
25mDeploying Python at Scale with Dask
44mUsing Dask for Parallel Computing in Python
30mDask ❤️ Xarray: Geoscience at Massive Scale
01h 03mDask: Next Steps in Parallel Python
29mScaling Data Workloads with Cluster Managers