Efficient shared memory data structures
summary
Claims that large read-only data can be efficiently shared across processes by placing C-like schemas into memory-mapped files and exposing lightweight proxies that index bytes without full deserialization. Implements schemas with fixed-size fields, relative pointers for movable objects, an identity map to avoid cyclic-serialization loops, and compact hash table indexes plus an exponential-then-binary search for cache-friendly lookups. Prototype uses struct/mmap and speeds attribute access by Cythonized proxies; it works on SSDs and spinning disks.
AI-written summary · about
chapters (10)
- 0:00Start
- 0:37Python
- 1:23Sharing memory... what for?
- 4:01Why not multiprocessing?
- 4:57The shared buffer
- 7:52Structs
- 10:56Proxies - adding complexity
- 15:37Manipulation without serialization
- 21:46Approximate associative maps
- 23:44Speed
related talks
50mInside PostgreSQL Shared Memory
01h 08mC++ and Persistent Memory Technologies, Like Intel's 3D-XPoint
43mImproving scalability; Reducing overhead in shared memory
52mChanging shared_buffers on the fly
01h 02m“Lightweight Object Persistence With Relocatable Heaps in Modern C++"
01h 13mC++ Memory Model Meets High-Update-Rate Data Structures
12mSegcache: a memory-efficient and scalable in-memory key-value cache for small objects
59mA Hash Map for Graph Processing Workloads & a Methodology for Transactional Data Structures