Parallel processing using CRDTs
summary
Claims that CRDTs let subprocess workers mutate local copies with automatic merge and low latency, demonstrated using PICRDT (wires/Yjs binding) and contrasted with HAMT-based Memhive. A benchmark on a 1,000,000-key map / 100 mutations shows CRDT updates far faster than pickling, though HAMT can beat CRDTs on very small datasets. Experiments compare HTTP vs memory pipes and client‑server vs peer‑to‑peer topologies, finding pipes and peer‑to‑peer scale better while server relay becomes the bottleneck.
AI-written summary · about
chapters (9)
- 0:00Parallel processing with CRDTs
- 1:11Python parallelization methods
- 3:44Memhive and HAMT for shared memory
- 8:00What are CRDTs
- 10:54Performance: HAMT vs pickle
- 14:08Transport and topology
- 19:10Scalability trade-offs
- 20:17Future: WireDB project
- 21:22Q&A
related talks
25mRayon: Data Parallelism for Fun and Profit
45mRethinking the Language Runtime for Scale
27mWhoops! I Rewrote It in Rust
41mCap'n Proto and Rust: Type Systems for Sharing
31mImmutable Data Structures and why You want them
28mScaling Real-Time: Building Federated Subscriptions in Rust
05mBuilding Efficient Web Scrapers
54mKEYNOTE: A journey through incremental computation