Make it Work, Make it Right, Make it Fast Debugging and Profiling in Dask
summary
Shows concrete before and after: a serial pandas run ~13s reduced to ~9s then to ~4s and finally ~2s by changing algorithm and scheduler. Demonstrates tools: memory_profiler, snakeviz and line profilers, IPython %time, PDB for local debugging, the Dask Dashboard for live traces, and how delayed + persist and swapping threaded → multiprocessing scheduler or distributed cluster cut I/O and GIL bottlenecks while keeping intermediates small.
AI-written summary · about
chapters (11)
- 0:00Dask introduction
- 3:10Live flight data coding
- 10:06Memory inefficiencies
- 14:30SnakeViz profiling
- 17:37Debugging with PDB
- 19:51Dask diagnostics tools
- 23:11Switching schedulers
- 25:01Distributed scheduler
- 28:02Scaling to cluster
- 29:51Persisting data
- 31:22Optimization summary
related talks
01h 26mDask tutorial
02h 18mHacking Dask: Diving Into Dask;s Internals
01h 03mDask: Next Steps in Parallel Python
01h 23mData Science at Scale with Dask
01h 22mUsing Dask for Parallel Computing in Python
44mDask for ad hoc distributed computing
44mUsing Dask for Parallel Computing in Python
30mHow to Tune your Python Analysis Pipeline: A Profiler Guide