Inside the UDS deduplication Index
summary
The UDS index used by dm‑vdo is a separate module that gives asynchronous advisory answers and requires vdo to verify any match, keeping correctness while avoiding write‑path blocking. The index is fixed‑size (0.25 GB–1 TB memory) with on‑disk storage ≈10× memory, yielding about 10 TB per GB of dedupe window; it stores 128‑bit signatures (currently MurmurHash3) and can use delta lists and sampling to compress/accelerate lookups, with optional larger 256‑bit cryptographic hashes for stronger collision resistance.
AI-written summary · about
chapters (7)
- 0:00Intro
- 3:18How does it work
- 5:35Memory footprint
- 7:19In memory pieces
- 10:17How does this work
- 12:48PR
- 15:46Spatial locality
related talks
40m"Gaining Constant time Lookup over Unorganized Data"
22mLoopDelta: Embedding Locality-aware Opportunistic Delta Compression in Inline...
25mSketching Volume Capacities in Deduplicated Storage
44mOpenMDM - Data Management for SDV
40mStorage Architecture for Yandex.Direct Advertising Objects
24mEfficient Similar-Object Search Algorithms for Terabytes of Data
07mdbverse: Composable database libraries for larger-than-memory scientific analytics
01h 00mMDS and DQS - Beyond the TLAs to Data Quality