Speed Up Your Data Processing
summary
Claims that parallel and asynchronous Python code speeds up pipelines and shows how. Uses pandas for in-memory work and the stdlib concurrent.futures with ThreadPoolExecutor for many IO-bound API requests (data.sg JSON) giving ~20.9× speedup: list comprehension 16.3 minutes versus 46.83 seconds with 40 threads. Uses ProcessPoolExecutor / multiprocessing for CPU-bound image resizing, cutting 29.6s to 6.98s on an 8‑core run (~4.3×). Notes limits: GIL and Amdahl overheads.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:1980-20 rule in data science
- 7:31Large datasets in pandas
- 10:43Small big data
- 11:28Parallel processing with toast
- 16:11Sync vs async with coffee
- 18:42Parallelism limits
- 24:23Multiprocessing vs multithreading
- 26:30concurrent.futures for tasks
- 27:56Parallel speedup case studies
- 29:46Key takeaways
related talks
01h 01mParallel Python Analyzing Large Data Sets
10mRedlining Your CPU for Fun and Profit: Threading and Multiprocessing in Python
30m5 Steps to Speed Up Your Data-Analysis on a Single Core
30mParallel computing in Python: Current state and recent advances
30mPython Serialization and the Hallows of Parallel Computing
35mWriting robust, readable, and maintainable concurrent programs in Python
30mParallelization of code in Python for beginners
02h 57mPython Concurrency: from beginner to pro