Sketching Volume Capacities in Deduplicated Storage

on YouTube ↗25m

" summary

Sketch-based estimators use content-defined sampling of chunk fingerprints (keep hashes whose leading bits meet a threshold, e.g. parameter 13) to represent a deduplicated system at tiny size and answer reclaimable capacity and attributed capacity queries, including cross-system migration cost. Algorithms track per-chunk volume references, apply scaled estimates from the sketch, and come with analytical error bounds (Chernoff-style) and empirical accuracy. Implementation runs off-box and is deployed on an IBM FlashSystem A9000.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 8:16Sketching
3. 11:39Notes
4. 15:03Deployment
5. 16:36Analysis
6. 18:57Data Center Optimization
7. 20:51Conclusion

" related talks

• 24mSketching Algorithms: Making Sense of Big Data in a Single StrokeTun Shwe · Conf42 Python 2024
• 01h 24mStreaming, Sketching and Sufficient Statistics IGraham Cormode · Big Data Boot Camp 2013
• 31mAnswering Multi-dimensional Queries on Massive DatasetsSrimathi H · Developer Summit 2020
• 01h 50mData sketching workshopWilliam Benton, Erik Erlandson · DevConf.CZ 2019
• 01h 13mSketching High-Dimensional DataJelani Nelson · Probability, Geometry, and Computation in High Dimensions Boot Camp (Simons Institute, Fall 2020)
• 01h 26mStreaming, Sketching and Sufficient Statistics IIGraham Cormode · Big Data Boot Camp 2013
• 47mData Sketches — How to Eat an Elephant Whole (Even If It's Infinite)Sergey Zhemzhitskiy · HighLoad++ 2023
• 01h 09mSketching High-Dimensional DataJelani Nelson · Probability, Geometry, and Computation in High Dimensions Boot Camp (Simons Institute, Fall 2020)