Downloading a Billion Files in Python
summary
Single-machine setup (16 cores, 64 GB) downloads one billion ~100‑byte files from a paginated REST API. Synchronous requests cost ~3 ms/request (~35 days). Multithreading adds overhead and is slower. Multiprocessing with concurrent.futures drops to ~0.32 ms/request (~3.7 days). Asyncio with aiohttp + uvloop sits between. A hybrid design uses processes each running an event loop plus a thread bridge, reduces IPC by sending pagination tokens instead of per-file tasks, and achieves ~30 μs/request (~8.4 hours). use multiprocessing for core parallelism hybrid (process+async+thread) yields best raw speed
AI-written summary · about
chapters (37)
- 0:00Introduction
- 0:29The Task
- 1:00Understanding the Task
- 1:23Network Latency
- 1:38File Size
- 2:10The API
- 3:08The Get API
- 3:30Disclaimers
- 4:38Synchronous
- 7:12Multithreading
- 8:48Coding
- 9:13Main Loop
- 9:56Performance
- 10:37Why is this happening
- 11:22Things to keep in mind
- 12:58Multiprocessing
- 13:45Multiprocessing code
- 14:48Iterating over pages
- 15:15Downloader
- 16:28Speed Improvements
- 17:31Async IO
- 17:46List Call
- 18:00Async IO Task
- 18:59Different Libraries
- 19:25Setup
- 20:07IO HTTP
- 20:32ItAll Files
- 20:49Download Files
- 22:36Summary
- 23:06Multi Processing
- 23:29Threading
- 23:51Workflow
- 25:57Interprocess communication overhead
- 26:25Combo results
- 26:43The real summary
- 27:01Lessons learned
- 27:39Thank you
related talks
30mSpeed Up Your Data Processing
29mParallelism Shootout: threads vs asyncio vs multiple processes
27m"Python’s tale of concurrency"
21mUse Python to process 12mil events per minute and still keep it simple (Talk)
37mCommon asynchronous patterns in Python
40mScaling Python - Bank Edition
30mParallelization of code in Python for beginners
34mUnderstanding of distributed processing in Python