Dask tutorial
summary
Shows Dask (local and distributed) used to parallelize Python: turning slow for‑loops into futures‑based parallel tasks, scaling pandas workflows to a 100 GB NYC taxi dataset split across 720 parquet files, and running Jupyter notebooks via Coiled. Demonstrates chunking/partitioning and why good chunksize matters, the Dask dashboard and task stream for performance, persist to memory vs repeated S3 reads, that drop_duplicates causes an expensive shuffle and can OOM, and that simple aggregations complete orders of magnitude faster.
AI-written summary · about
chapters (10)
- 0:00Dask tutorial intro
- 3:43What Dask is and how it works
- 9:27Setup tutorial environment
- 26:12Local Dask DataFrame basics
- 37:09Dask Futures for parallel Python
- 49:00Scaling to distributed cluster
- 55:05Large datasets with Dask DataFrame
- 1:03:16Advanced Dask operations
- 1:16:50Custom functions and Dask API
- 1:25:17Q&A
related talks
01h 03mDask: Next Steps in Parallel Python
01h 22mUsing Dask for Parallel Computing in Python
44mDask for ad hoc distributed computing
44mUsing Dask for Parallel Computing in Python
01h 23mData Science at Scale with Dask
01h 56mParallelizing Scientific Python with Dask
02h 18mHacking Dask: Diving Into Dask;s Internals
01h 17mScaling PyData with Dask