Using Dask for Parallel Computing in Python
summary
Claims that Dask scales numpy and pandas via familiar structures and delayed task graphs and shows working examples: a billion 32‑bit floats written as 4 GB on disk (HDF5), computing a mean with NumPy taking ~16 seconds vs 7 seconds with Dask while using megabytes of RAM; demos of dask.array, dask.bag, dask.dataframe (a collection of pandas frames), use of threaded/process/distributed schedulers, Jupyter notebooks, and real UK food‑survey results (milk highest), all executed end to end.
AI-written summary · about
chapters (16)
- 0:00Dask introduction and setup
- 3:01What is Dask and why use it
- 6:12Dask computation and scheduling
- 8:02Dask arrays and NumPy interface
- 13:19Compute and task graphs in Dask
- 19:15Scheduling backends in Dask arrays
- 26:12HDF5 files and blocked algorithms
- 32:42Dask bags and functional primitives
- 41:15Semi-structured data with Dask bags
- 47:05Fold by and group by in Dask bags
- 52:06Exercise: total amounts with Dask bags
- 59:57Dask data frames introduction
- 1:02:27Dask data frames and pandas API
- 1:07:45Reading UK survey data
- 1:13:27Dask data frame methods and map partitions
- 1:17:16Analyzing UK food consumption trends
related talks
44mUsing Dask for Parallel Computing in Python
01h 24mData Science at Scale with Dask
01h 56mParallelizing Scientific Python with Dask
01h 03mDask: Next Steps in Parallel Python
01h 26mDask tutorial
01h 23mData Science at Scale with Dask
25mDeploying Python at Scale with Dask
01h 17mScaling PyData with Dask