Overcoming GIL with subinterpreters and immutability
summary
A new framework called memhive implements a subinterpreter worker pool, an efficient RPC layer and an async/await API and claims to unlock true parallelism without serializing large datasets. It uses immutable collections built with an HT/Hashed Array Mapped Trie style structure and structured sharing so mutating a million‑entry mapping copies only a few nodes, not the whole object. Benchmarks show submillisecond transfers versus pickle and large speedups when few entries change; implementation requires remote refcount tricks, lazy memcpy of scalars, and is a prototype on GitHub (Apache‑2).
AI-written summary · about
chapters (8)
- 0:00Welcome to subinterpreters
- 3:31Python GIL limits
- 7:03Immutability for shared state
- 10:15Efficient immutability with HAMT
- 14:27Sharing immutable data
- 18:04Benchmarking sharing vs pickling
- 21:53Mamihe state and future
- 22:56Q&A on use cases
related talks
44mWorking Around the GIL with asyncio
30mA Per-Interpreter GIL: Concurrency and Parallelism with Subinterpreters
23mParallelization of Dynamic Languages - Synchronizing Built-in Collections
38mDemystifying the GIL
31mThinking Outside the GIL with AsyncIO and Multiprocessing
32mDesigning With Immutability
46mImmutability Against The Machine
26mSingle node shared memory comes to dask