Parallelization of code in Python for beginners
summary
Shows concrete recipes: use joblib with a number-of-workers ≈ physical CPU cores and tune chunk size, prefer multiprocessing for blackbox Python code, beware duplicating large inputs across processes. Demonstrates regex text cleaning on the Amazon Fine Food reviews dataset (~500k rows) where a single-thread run hit ~13 minutes and a 4-core joblib run drops to ~6 minutes. Also parallelizes 54 hyperparameter jobs (≈1 minute each) from ~60 to ~30 minutes and explains numpy/pandas avoids copying large arrays.
AI-written summary · about
chapters (9)
- 0:00Introduction to parallelization
- 1:50When parallelization applies
- 6:08Multi-threading vs multi-processing
- 10:04CPU cores and memory usage
- 14:17Multi-processing over threading
- 18:57Text pre-processing with Joblib
- 26:08Hyperparameter tuning in parallel
- 27:52Parallelization gotchas and tips
- 29:16Summary
related talks
30mParallel computing in Python: Current state and recent advances
34mUnderstanding of distributed processing in Python
30mSpeed Up Your Data Processing
01h 01mParallel Python Analyzing Large Data Sets
10mRedlining Your CPU for Fun and Profit: Threading and Multiprocessing in Python
41mParallelism in Numerical Python Libraries
30mPython Serialization and the Hallows of Parallel Computing
01h 56mParallelizing Scientific Python with Dask