Data Science at Scale with Dask
summary
Introductory walkthrough of Dask shows concrete wins: scaling pandas and NumPy to disk‑heavy workloads (12 CSVs ~15 GB, an example dataset of 665 million rows), using lazy evaluation so operations build task graphs until a compute call, and running code on Coiled clusters (example: 20 workers with 16 GB each) or local distributed clients. Demonstrations include dd.read_csv partitioning, map_partitions, dask.delayed for parallel loops (turning multi‑second serial sleeps into ~1s parallel), dask-ml and dask‑xgboost integrations, plus the dashboard and GitHub repo.
AI-written summary · about
chapters (15)
- 0:00Start
- 18:53Das Scheduler
- 30:43Spin Up a Coiled Cluster
- 38:53Schedulers
- 42:57Load in a Das Data Frame
- 50:07Caching an Intermediate Operation
- 52:01Examples
- 54:47Distributed Scheduler
- 58:28Machine Learning Notebook
- 59:15Overview of Scikit-Learn
- 59:26How Psychic Learn Works
- 1:00:07Hyper Parameter Optimization
- 1:07:40Das Ml
- 1:08:24Linear Regression Model
- 1:10:03Xgboost
related talks
01h 23mData Science at Scale with Dask
01h 22mUsing Dask for Parallel Computing in Python
01h 03mDask: Next Steps in Parallel Python
29mScaling Data Workloads with Cluster Managers
03h 07mData of Unusual Size: Interactive Visualization
02h 18mHacking Dask: Diving Into Dask;s Internals
18mUsing Dask to Distribute Machine Learning Jobs
04mDask Powering lower end PC