Dat
summary
Dat claims to be git for data, implemented in Node with LevelDB, protocol buffers and an abstract-blob-store, and it actually versions rows, streams changes, and supports large binary blobs on disk. The demo imports ~91,000 rows of a genome TSV, mirrors npm metadata with 117,000 modules and about 150 GB of tarballs, and runs pipelined processing via Docker and headless browsers. Multimaster replication, live pulls and a searchable registry are under active development.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:38Max Ogden @maxogden
- 1:23dat is an open source tool for sharing and collaborating on data
- 1:34we are grant funded and 100% open source
- 2:45reproducible science
- 4:21analogy time: lets talk about source control
- 4:43i want to fix a bug in cool-project
- 5:58claim: currently data sharing is a mess
- 6:34email csv files
- 6:47we want to do for data what git did for source code
- 12:23a data set we can all relate to: npm
- 14:30calculate how big npm is using dat
- 15:31transform the npm data using bulk-markdown-to-png
- 19:10bionode bioinformatics tools on npm
- 19:31gasket is a cross platform pipeline manager
- 20:10datscript is an experimental pipeline config language
- 20:39branches, dat checkout 3b2d98V3, multi master replication, sync to databases, registry
related talks
43mVersioning, Syncing & Streaming Large Datasets Using DAT + Node
25mManaging changes to your data with dat
35mGit: the stupid NOSQL database
39mGit the NoSQL Database
29mData Stored Everywhere and Nowhere - Building Applications on the Distributed Web
43mGit for Data- Data Versioning for Reproducible Data Science with Dolt
08mA Git-like Repository for Your Data Lake
40mUsing Git as a NoSql Database