Memory mapped storage
summary
Yandex Maps uses memory-mapped files to serve dozens-to-hundreds of gigabytes (examples cite ~100 GB) so reads hit almost zero read time when OS cache is warm; serialization is a two-pass writer that lays out var-size payloads then fixed headers, stores pointers as relative offsets, and exposes STL-like read-only types via reinterpret_cast. Benchmarks show traversal is within ~10% of in-memory STL speed, deserialization latency is essentially eliminated, but serialization is ~3× slower, files are larger, and format is non‑extensible and architecture‑dependent.
AI-written summary · about
chapters (27)
- 0:00Intro
- 0:50Who we are
- 1:30Data usage patterns
- 4:00Data reading
- 5:33Serialization: a usual approach
- 6:53The same thing in RAM
- 7:50Reconstructing the layout
- 8:26Storing pointers
- 10:26A string approaches
- 11:32A kind of duplicity
- 12:14Templates to the rescue!
- 13:27Key concepts (on a single slide)
- 14:37What can be serialized
- 15:14Two passes
- 16:38User-defined structs
- 17:35Pitfalls
- 18:54Containers
- 19:30vector
- 21:29deque, list, rope, slist...
- 22:18map, set
- 23:17unordered map, unordered set
- 25:48Downsides
- 28:01Data versions
- 29:22What we measure
- 30:24Outcome
- 32:23What we've got
- 33:20Possible use cases
related talks
56mC++ Relocation - How to Achieve Blazing Fast Save and Restore and More!
56mEverything is serialization
23mA Faster Serialization Library Based on Compile-time Reflection and C++ 20
01h 00mKilling C++ Serialization Overhead & Complexity
57mDynamic memory allocation challenges in safety critical systems
25mVector Streaming: The Memory Efficient Indexing for Vector Databases
26mAllocating Less: Really Thin Rust Cloud Apps
32mThe Impact of Memory Allocators on Performance: A Deep Dive