Understanding of distributed processing in Python
summary
Clear comparisons and concrete recommendations: threading hits the GIL blocks CPU threads limit so it’s only good for I/O, multiprocessing works for CPU-bound tasks but incurs copy and scheduling overhead, and joblib simplifies local parallelism. Empirical thresholds given (pandas most efficient under ~1GB; Dask scales across 1–4/5 machines for dataframe/array workloads; Spark targets Hadoop/cluster or terabyte‑scale jobs). Notes include task queues (Celery style), monitoring UIs, and when overhead makes distribution slower than a single node. Slides linked in uploader notes.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:10Cookpad and its services
- 2:41Agenda and objectives
- 3:46Parallel vs distributed processing
- 8:13Parallel vs distributed differences
- 10:09Python parallel libraries
- 16:45Joblib for parallel processing
- 19:24Distributed processing intro
- 20:06Celery for distributed tasks
- 22:46Dask for pandas processing
- 25:20PySpark for big data
- 28:50Avoiding distributed processing
- 29:34Recommendations and closing
- 30:18Q&A on MPI
related talks
30mParallel computing in Python: Current state and recent advances
30mParallelization of code in Python for beginners
31mPython Multithreading and Multiprocessing: Concurrency and Parallelism
35mWriting robust, readable, and maintainable concurrent programs in Python
30mPython Serialization and the Hallows of Parallel Computing
01h 22mUsing Dask for Parallel Computing in Python
30mSpeed Up Your Data Processing
31mGetting Around the GIL: Parallelizing Python for Better Performance