Thinking Outside the GIL with AsyncIO and Multiprocessing
summary
Combines AsyncIO and multiprocessing to sidestep the GIL by running an event loop inside child processes, using multiprocessing queues, batching and optional spawned start to cut copy‑on‑write memory. Benchmarks show naive multi+async ≈3.5×, tuned chunking ≈5–5.5× and a 8x speed-up when reframed as MapReduce-style aggregation, with a dramatic memory drop; implementation handles hundreds if not thousands of concurrent IO requests, is open sourced under MIT and proves the approach works in production.
AI-written summary · about
chapters (12)
- 0:00Start
- 0:50What's the GIL?
- 1:24Stateful monitoring
- 3:22#impact
- 4:33Not aging well
- 5:20Switch to py3
- 6:23Sharding
- 10:44Multiprocessing + Asynclo
- 15:41Optimizations
- 17:09Considerations
- 18:45Performance comparison
- 21:07aiomultiprocess
related talks
44mWorking Around the GIL with asyncio
32mEnhancing AsyncIO for Complex Applications
30mSpeed Up Your Data Processing
30mParallel computing in Python: Current state and recent advances
31mPython Multithreading and Multiprocessing: Concurrency and Parallelism
35mWriting robust, readable, and maintainable concurrent programs in Python
01h 21mTUTORIAL: Async Python, Good it is
34mA practical guide to speed up your application with Asyncio