Delta for your Data: How to efficiently track and store deeply nested data changes
summary
DeepDiff claims a production-grade way to version nested data: 1.7 million monthly downloads, a new Delta/patch that applies to CSV JSON YAML and arbitrary Python objects, and command-line tools to create and apply patches. It implements ignore-order, significant-digits rounding to suppress noise, a cache-tuned algorithm for large nested compares, and a deep distance similarity score 0–1 (lower is more similar). Patches serialize changes and successfully apply across formats; the library is used at scale by companies like Google and Netflix.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:40Existing solutions
- 6:04DeepDiv
- 9:26Bounce Rate
- 10:52Exclude Path
- 22:43Applications
- 24:38DeepGrip
- 27:43Conclusion
related talks
29mData Diffing Based Software Architecture Patterns
16mThe Evolution of Delta Lake
24mData Versioning Explained the Open Source Way
22mLoopDelta: Embedding Locality-aware Opportunistic Delta Compression in Inline...
11mIntroducing Dat: If Git Were Designed For Big Data
32mDesigning Data Infrastructure in the Age of Generative AI
39mDelta Lake: Diving into Data Lakes Without the Downsides
21mBuilding Data Integration Services In Real Time