Improving scalability; Reducing overhead in shared memory
summary
Work targets big shared structures so autoscaling can shrink shared_buffers without hitting immovable metadata. It replaces dino_hash with co‑allocated buckets and elements and uses offset pointers instead of full pointers, cutting hash element overhead to ~12 bytes plus payload. Buffer tag goes from 20 to 8 bytes, checkpoint-sort items 20→12, and per-buffer shared-memory cost drops from ~150 bytes to ~116 bytes (+16 bytes for fork entries). Lock tables shrink by about 30% in tests and lookup cache‑misses should fall.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:17Problem and goals
- 3:51Optimization strategies
- 6:01Buffer manager overview
- 8:53New hash table design
- 18:13Reducing buffer table size
- 24:22Memory savings summary
- 25:28Lock manager structure
- 31:05Optimizing lock tables
- 33:17Space savings and impact
- 39:48Performance and NUMA considerations
- 42:45Q&A and conclusion
related talks
27mEfficient shared memory data structures
14mSEPH: Scalable, Efficient, and Predictable Hashing on Persistent Memory
25mMemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing
45mEnhancing PostgreSQL Plasticity: New Frontiers in Memory Management
01h 07mEmerging Embdedded Memories and Storage
24mPerformance tuning in BlueStore & RocksDB
23mSize-aware Sharding For Improving Tail Latencies in In-memory Key-value Stores
37mMemtable Pluggability and Alternate Memtables (CEP-11 & CEP-19)