Scaling Data Workloads with Cluster Managers
summary
Shows how the same Python notebook can run unmodified from a laptop to large clusters by using Dask (and Ray) as parallel backends with familiar DataFrame APIs (Pandas/Modin/Polars) and Coiled for cloud orchestration. Explains deployment patterns: either a cloud engineer deploys a cluster and exposes a load‑balancer endpoint or an analyst SSH‑tunnels to an EMR/YARN head, and notes ecosystem issues—Dask Cloud Provider is broken and Dask‑Yarn only works with older 2021 combos.
AI-written summary · about
chapters (7)
- 0:00One-shot data science
- 3:04Data analyst vs engineer
- 7:30Dask vs Ray for distributed
- 11:10Scaling Python with Dask
- 16:57Dask on Kubernetes
- 19:28Dask demo local/cloud
- 23:43Best practices
related talks
01h 17mScaling PyData with Dask
25mDeploying Python at Scale with Dask
01h 24mData Science at Scale with Dask
01h 03mDask: Next Steps in Parallel Python
01h 23mData Science at Scale with Dask
18mUsing Dask to Distribute Machine Learning Jobs
01h 22mUsing Dask for Parallel Computing in Python
02h 18mHacking Dask: Diving Into Dask;s Internals