Python Serialization and the Hallows of Parallel Computing
summary
Compares threading vs multiprocessing with concrete timings: 16 URLs: serial 6.0s, 4 threads 2.1s, 4 processes 2.8s shows threads win for IO due to lower spawn overhead, while CPU counter: serial 4.2s, 4 threads slower, 4 processes 1.9s shows processes win for CPU-bound work. Explains pickle serializes by name so lambdas and functions defined in IPython fail, and shows solutions: dill/cloudpickle serialize by value but are slower; an in-house “disco pickle” uses CPython C pickler for ~100× speedups and avoids global dispatch overrides, with tradeoffs around security and code-change consistency.
AI-written summary · about
chapters (16)
- 0:00Introduction to talk
- 1:51What is parallel computing
- 3:37Threading vs multiprocessing
- 5:45CPU-bound vs IO-bound tasks
- 6:49How multiprocessing works
- 8:12Pickling limitations in Python
- 10:01Pickling in Jupyter/IPython
- 11:03Serialization by value
- 13:06Trade-offs in pickling
- 14:31Introducing disco pickle
- 17:44Deshaw’s pickle optimizations
- 18:13Q&A session
- 22:21Async programming and parallelism
- 24:59Async vs threading benefits
- 27:57Concurrency vs parallelism
- 29:46Closing remarks
related talks
30mParallel computing in Python: Current state and recent advances
30mSpeed Up Your Data Processing
35mWriting robust, readable, and maintainable concurrent programs in Python
34mUnderstanding of distributed processing in Python
39mUnlocking the Parallel Universe: Subinterpreters and Free-Threading in...
31mPython Multithreading and Multiprocessing: Concurrency and Parallelism
29mParallelism Shootout: threads vs asyncio vs multiple processes
30mParallelization of code in Python for beginners