Using Dask for Parallel Computing in Python

on YouTube ↗44m

" summary

Dask provides parallel arrays, bags, and dataframes that build task graphs from many small NumPy or pandas pieces, run on threads, processes, or a distributed cluster, and execute from Jupyter notebooks. The demo builds a 4 GB HDF5 random array, compares NumPy (~16 s, single thread) to Dask (~3 s, parallel) and shows low RAM use, task visualizations and a progress bar. Exercises use chunking, HDF5, CSV partitions and confirm the approach works.

AI-written summary · about

" chapters (8)

1. 0:00Dask and parallel computing
2. 6:14Dask collections and task graphs
3. 17:46Scheduling backends
4. 21:37Large arrays on disk
5. 27:37NumPy performance comparison
6. 32:35Dask bag for JSON
7. 34:23Dask data frames and pandas
8. 42:39Final notes

" related talks

• 01h 22mUsing Dask for Parallel Computing in PythonSkipper Seabold · PyData Chicago 2016
• 01h 03mDask: Next Steps in Parallel PythonMatthew Rocklin · PyCon.DE 2017
• 01h 26mDask tutorialPatrick Hoefler · PyData NYC 2023
• 01h 56mParallelizing Scientific Python with DaskJim Crist, David Mertz · PyData Seattle 2017
• 01h 23mData Science at Scale with DaskRichard Pelgrim · PyCon DE & PyData Berlin 2022
• 01h 17mScaling PyData with DaskMartin Durant, Jim Crist · PyData DC 2018
• 25mDistributed data workflows: PySpark vs DaskVaibhav Srivastav · DevConf.CZ 2020
• 38mScalable Machine Learning with DaskTom Augspurger · PyData New York City 2019