Dask: Next Steps in Parallel Python

on YouTube ↗01h 03m

" summary

Dask demonstrates running familiar pandas code on a cluster by breaking a 20 gigabytes on disk dataset into ~300 blocks, executing on a 64 cores cluster with a scheduler and workers, showing interactive groupbys and a computed 38% tip at 4:00 a.m.; it builds task graphs via dask.delayed, exposes a dashboard, offers Dask dataframe and Dask array, plugs into scikit-learn/joblib and XGBoost, and includes geo and streaming experiments with measurable speedups and real data transfer and scheduling details.

AI-written summary · about

" chapters (6)

1. 0:00Start
2. 20:50Task Graphs: Pipelined Grid Search
3. 34:22Dask Interfaces
4. 40:12Machine Learning: We have a few options ...
5. 44:05Use Dask Array to Build Optimization Algorithms
6. 47:25Deploy Other Services with Dask

" related talks

• 01h 26mDask tutorialPatrick Hoefler · PyData NYC 2023
• 01h 22mUsing Dask for Parallel Computing in PythonSkipper Seabold · PyData Chicago 2016
• 01h 24mData Science at Scale with DaskRichard Pelgrim · PyData London 2022
• 01h 56mParallelizing Scientific Python with DaskJim Crist, David Mertz · PyData Seattle 2017
• 01h 23mData Science at Scale with DaskRichard Pelgrim · PyCon DE & PyData Berlin 2022
• 44mUsing Dask for Parallel Computing in PythonMatthew Rocklin · PyData DC 2016
• 01h 17mScaling PyData with DaskMartin Durant, Jim Crist · PyData DC 2018
• 44mDask for ad hoc distributed computingMatthew Rocklin · PyData DC 2016