Managing changes to your data with dat
summary
A git-like system for datasets that actually handles terabytes by storing a Merkle DAG of content-addressed blobs, using leveldb + abstract-leveldown for pluggable on-disk backends and doing streaming replication over HTTP websockets or SSH. No merge conflicts — concurrent edits become forks that can be merged by adding parent links. Replication first exchanges hashes then transfers only missing blobs via resumable streams; prototypes target astronomy terabyte pilots and have bit‑torrent and Google Sheets integration plans.
AI-written summary · about
chapters (15)
- 0:00what the fork is dat?
- 0:27ALFRED P. SLOAN FOUNDATION
- 3:51How do scientists reproduce research?
- 5:16git is for source code, and thats ok.
- 5:32How do you build a frictionless git- like tool for data?
- 6:35AbstractLEVELDown
- 11:15Merkle Directed Acyclic Graph
- 13:21But what about conflicts?
- 16:11in dat, each graph node points to the data storage.
- 17:38Replication
- 17:49computer 1 wants to push new merge to computer 2
- 18:44How does computer 2 know which ones are missing?
- 18:59Binary search
- 19:13It doesn't matter.
- 22:16Implications for open data
related talks
22mDat
43mVersioning, Syncing & Streaming Large Datasets Using DAT + Node
29mData Stored Everywhere and Nowhere - Building Applications on the Distributed Web
27mRevision Control for Structured Data
08mA Git-like Repository for Your Data Lake
36mData Versioning - does it mean what we think it means?
43mGit for Data- Data Versioning for Reproducible Data Science with Dolt
35mData versioning in machine learning projects