Common asynchronous patterns in Python
summary
Problem: crawl a list of referrer URLs, extract links, visit each link and grab page titles. A straight synchronous loop takes about nine minutes for 60 sample sites. Thread pools don’t help because the work is IO bound; switching to event libraries—gevent, Twisted, and especially async IO with aiohttp—cuts wall time (rounded to under 3 minutes in demo). Key wins come from streaming HTML parsing and breaking work into small, testable stages wired by queues so links are queued as they’re found and consumers run concurrently.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:53The problem
- 4:48Trying threading
- 6:56Understanding I/O bound
- 9:01Using Gevent
- 10:33Using Twisted
- 14:36Using Asyncio
- 16:55Architecting with queues
- 21:06Streaming data
- 24:19Key takeaways
- 25:19Q&A
related talks
27m"Python’s tale of concurrency"
29mParallelism Shootout: threads vs asyncio vs multiple processes
56mLeveraging Async and Await in Modern Python
30mLet's get you started with asynchronous programming
32mDemystifying AsyncIO: Building Your Own Event Loop in Python
34mA practical guide to speed up your application with Asyncio
35mHow Do Python Coroutines Work?
28mWhy Should Async Get All The Love Advanced Control Flow With Threads