Parallel Python Analyzing Large Data Sets
summary
Shows concrete patterns and results: using concurrent.futures process/thread pools to turn a JSON-to-HDF5 conversion from slow sequential loops into a parallel map that finishes in seconds, a toy sleep loop that drops from ~8s to ~1s, and a cross‑validated hyperparameter search lifted to a 4‑node Google Compute Engine cluster launched from bigfatintegral.net. Claims and tradeoffs are explicit: map solves ~80% of cases, submit for arbitrary task graphs, and practical tools named like dask distributed executor, Spark, pandas, HDF5 and scikit‑learn; the cluster setup works.
AI-written summary · about
chapters (8)
- 0:00Parallel computing intro
- 4:48Three parallel paradigms
- 12:18Parallelizing with map
- 20:33Submit for unstructured parallelism
- 31:34Spark and Dask for big data
- 38:21Hyperparameter search in ML
- 54:07Dask distributed data frames
- 59:54Q&A
