Parallelizing Scientific Python with Dask
summary
Dask claims to parallelize familiar NumPy/Pandas workflows across cores or a cluster by building task graphs and running them on schedulers. Examples show wrapping functions with dask.delayed turns an eight‑second sequential loop into a one‑second parallel run on eight cores, and reading ten CSVs drops from ~6.5s to ~2s using dask.dataframe plus the distributed scheduler. Tools demonstrated include Pandas, NumPy, dask-scheduler/dask-worker and Client, with a diagnostics UI; notebooks run locally or on Google/Kubernetes and require conda/Anaconda.
AI-written summary · about
chapters (15)
- 0:00Parallelizing with Dask
- 5:20Setting up the notebook environment
- 6:02Dask delayed fundamentals
- 14:22Parallelizing simple computations
- 27:23Debugging delayed issues
- 37:40Real-world data processing
- 53:43Dask dataframes intro
- 1:01:40Parallel operations with dataframes
- 1:14:06Pandas-style exercises
- 1:23:25Dask schedulers
- 1:37:15Distributed clusters
- 1:42:38Dask ecosystem & ML
- 1:46:19Dask vs Spark
- 1:50:50Future plans & integration
- 1:54:54Q&A
related talks
01h 22mUsing Dask for Parallel Computing in Python
01h 03mDask: Next Steps in Parallel Python
44mUsing Dask for Parallel Computing in Python
01h 23mData Science at Scale with Dask
01h 26mDask tutorial
01h 17mScaling PyData with Dask
44mDask for ad hoc distributed computing
38mScalable Machine Learning with Dask