Python's Parallel Programming Possibilities - 4 levels of concurrency
summary
Compares four concrete concurrency levels with code examples: multiple machines via RQ with Redis (jobs pushed as Redis lists, demonstrated across hosts/VMs), multiprocessing/threads using Python stdlib (forking, joinable queues, GIL limits shown with numpy vs pure Python), and asyncio + aiohttp (async HTTP fetches, await text, gather). Shows arq — an asyncio+Redis job queue/RPC that retries failed jobs and runs reliably in production.
AI-written summary · about
chapters (8)
- 0:00Introduction to concurrency
- 1:25Multiple machines
- 7:14Multiple processes
- 10:29Multiple threads
- 14:34Async I/O
- 19:17Combining concurrency models
- 20:47Summary and best practices
- 22:17Q&A and closing remarks
related talks
31mPython Multithreading and Multiprocessing: Concurrency and Parallelism
35mWriting robust, readable, and maintainable concurrent programs in Python
29mManaging scientific simulations with Python with RQ (Redis Queue)
02h 57mPython Concurrency: from beginner to pro
29mParallelism Shootout: threads vs asyncio vs multiple processes
27m"Python’s tale of concurrency"
56mLeveraging Async and Await in Modern Python
30mParallelization of code in Python for beginners