Twisted up in a Distributed Tornado
summary
Begins by showing concrete exercises not platitudes a t SNE image pipeline pulling images then parallelizing with process and thread pools and scaling to Dask/Distributed the notebook pipeline produces a usable scatterplot the streaming Wikimedia example writes into MongoDB via Motor and a simple Tornado WebSocket dashboard streams updates successfully a benchmark runs 3,600 parallel tasks sleeping and finds thread pools sometimes faster explains that cancellation is unreliable and recommends a timeout decorator or Luigi for killable tasks and claims a six month path from novice to Dask contributor
AI-written summary · about
chapters (11)
- 0:00Async Python at CigarMe
- 5:54Concurrency and futures
- 12:25Image processing with executors
- 15:46Flow control in async
- 21:15Async IO vs Tornado coroutines
- 29:13Non-blocking HTTP in Tornado
- 39:47Async HTTP with aiohttp
- 43:28Task timeouts in async Python
- 51:10Scaling async workflows
- 58:19Jupyter WebSockets
- 1:04:03Summary and async apps
related talks
01h 26mDask tutorial
02h 18mHacking Dask: Diving Into Dask;s Internals
01h 03mDask: Next Steps in Parallel Python
01h 23mData Science at Scale with Dask
37mMake it Work, Make it Right, Make it Fast Debugging and Profiling in Dask
44mUsing Dask for Parallel Computing in Python
01h 22mUsing Dask for Parallel Computing in Python
44mDask for ad hoc distributed computing