DupHunter: Flexible High-Performance Deduplication for Docker Registries

on YouTube ↗22m

" summary

DupHunter is a Docker registry architecture that natively deduplicates unpacked layer files, stores unique file chunks across servers as layer slices and slice recipes, and uses a two‑tier pre-construct/pre‑fetch cache plus parallel slice reconstruction to avoid expensive layer restores. It implements configurable modes (keep n intact replicas or selective S‑mode by popularity), evaluates with IBM production traces and ~1 TB of image data, and demonstrates up to 6.9x storage reduction and up to 2.8x GET latency improvement.

AI-written summary · about

" chapters (17)

1. 0:00Intro
2. 0:37Containers are ubiquitous
3. 1:05Application containerization is becoming a significant market player
4. 2:16Our contribution: DupHunter-a framework to deduplicate images in Docker registries
5. 6:00Key observation II: Predictable user access pattern We observe a consistent user pulling pattern: Pull manifest first, then layers, but not all of the layers will be pulled
6. 6:53Key observation Il-b: User repulling pattern can also be predicted
7. 8:11Key observation Il-c: Layer preconstruction is possible
8. 9:20DupHunter architecture
9. 9:54Reducing overhead in DupHunter
10. 10:17DupHunter supports multiple replica deduplication modes
11. 12:57Deduplicating layers
12. 13:53Restoring layers
13. 15:28Cache handling in tiered storage
14. 16:36Evaluation
15. 18:08Deduplication ratio vs. performance
16. 19:30Prefetch cache hit ratio
17. 20:17Preconstruct cache hit ratio