Dask for ad hoc distributed computing
summary
Describes a pure-Python dynamic task scheduler that executes arbitrary task graphs, coordinates workers and a central scheduler, and measures runtimes and byte sizes to make locality and placement decisions. Shows Dask Array and Dask DataFrame on 28 GB NYC taxi data in ~3 s for a groupby, reports ~200 microsecond per-task overhead and ~5,000 tasks/sec throughput, exposes a web UI and task visualizations, supports single-machine and cluster modes, handles worker failure and dynamic scaling, and serializes code with cloudpickle so custom Python functions run unchanged.
AI-written summary · about
chapters (7)
- 0:00Dask introduction
- 6:42Single machine vs cluster
- 11:16Custom computations with Dask
- 17:35Dynamic task scheduler
- 29:54Dask vs Spark
- 34:01Handling failures
- 39:23Distributed scheduler deep dive
related talks
01h 26mDask tutorial
01h 03mDask: Next Steps in Parallel Python
01h 22mUsing Dask for Parallel Computing in Python
02h 18mHacking Dask: Diving Into Dask;s Internals
01h 23mData Science at Scale with Dask
01h 56mParallelizing Scientific Python with Dask
25mDistributed data workflows: PySpark vs Dask
38mScalable Machine Learning with Dask