Parallelism Shootout: threads vs asyncio vs multiple processes
summary
Downloads lots of URLs from a file and implements and times four concrete solutions: plain sequential loop, many threads backed by a Queue, a multiprocessing approach (including Pool), and a single-process asyncio coroutine solution using yield from and an event loop. Benchmarks use 30 URLs: sequential ~30s, threads drop to ~5s, multiprocessing similar, and asyncio scales best across larger URL sets. Code examples show the queue, Pool.map, coroutine structure and measured timings; everything runs and compares predictably.
AI-written summary · about
chapters (13)
- 0:00Parallelism Shootout threadsve multiple processes asyncio
- 1:16Using three different modules: threading, multiprocessing and asyncio.
- 1:44Break down the problem.
- 2:01Before we begin...
- 3:02Before we parallelise...
- 5:27Actual real POSIX threads (pthreads) or Windows threads.
- 11:12threading example to multiprocessing...
- 11:42Convenient means of parallelising the execution of a function across multiple input values, distributing the input data across processes (data parallelism).
- 11:59sequential example to multiprocessing...
- 14:25Provides infrastructure for writing single- threaded concurrent code.
- 14:58Basic asynio Concepts
- 16:50The component that is in charge of keeping track of and scheduling all the coroutines that want time on the thread.
- 19:47Drum roll
related talks
27m"Python’s tale of concurrency"
37mCommon asynchronous patterns in Python
32mPython's Parallel Programming Possibilities - 4 levels of concurrency
24mA Crash Course in Python 3's asyncio
56mLeveraging Async and Await in Modern Python
30mPython Serialization and the Hallows of Parallel Computing
41mCoroutines, Async/Await, Asyncio and the Pulsar Library
25mThe hitchhiker's guide to asyncio