Hacking Dask: Diving Into Dask;s Internals
summary
Shows Dask internals with runnable specifics: task graphs are created and optimized (fuse, inline, cull), the distributed scheduler and workers run via LocalCluster and client, diagnostics and a saved performance report HTML capture task streams, and a scheduler and worker plugin APIs let code run at lifecycle events (example: pip-install plugin). Demonstrations include a 16.2 GB array completed on 16 GB laptop and a loop sped from ~5.5s to 1.77s using delayed/futures, proving the techniques work.
AI-written summary · about
chapters (40)
- 0:00Start
- 0:37Overview
- 0:54Overview Notebook
- 2:15Why Would You Want To Use Desk
- 3:59Das Collections
- 5:29Task Graphs
- 7:23Desk Arrays
- 11:53Blocked Algorithms
- 15:15Das Data Frame
- 20:55Schedulers
- 21:32Single Machine Schedulers
- 22:50Swap between the Different Schedulers
- 23:14Set the Default Scheduler
- 26:31Futures Interface
- 30:49Custom Operations
- 32:07Block Computations
- 33:46Task Graph
- 35:19Map Blocks
- 38:21Special Arguments
- 38:40Output
- 46:23Diagnostic Dashboards
- 49:33Map Partitions
- 58:48Reduction
- 1:01:15Group by Aggregation
- 1:04:43Graph Optimizations
- 1:11:49Fuse Function
- 1:13:40Graph Chain
- 1:26:49Distributed Scheduler
- 1:27:50Distributed Cluster Overview
- 1:28:06Scheduler
- 1:30:44Client Process
- 1:34:14Client Restart Method
- 1:36:01Deploying Das Clusters
- 1:41:18Create a Cluster
- 1:41:57Cluster Api
- 1:47:06Inspecting the State of the Cluster
- 1:48:40Spin Up a Local Cluster
- 1:50:35Performance Report
- 1:51:18Performance Report
- 1:54:59Scheduler Python Object
related talks
01h 03mDask: Next Steps in Parallel Python
01h 26mDask tutorial
44mDask for ad hoc distributed computing
01h 23mData Science at Scale with Dask
01h 24mData Science at Scale with Dask
37mMake it Work, Make it Right, Make it Fast Debugging and Profiling in Dask
01h 22mUsing Dask for Parallel Computing in Python
44mUsing Dask for Parallel Computing in Python