Versioning, Syncing & Streaming Large Datasets Using DAT + Node

on YouTube ↗43m

" summary

Dat is a usable system that actually version-controls and replicates datasets: a streaming dataset versioning + replication tool built on Node with LevelDB + blob stores and a command-line API. Demonstrations show a skimmed npm metadata clone of ~113k modules and a full npm clone sized over 100+ GB, lazy-fetching tarballs on demand, and ingestion of terabyte-scale CSVs and even ~40 TB raw imagery for astronomy. Live pulls, Docker pipeline execution, and sync-to-databases run end to end.

AI-written summary · about

" chapters (27)

1. 0:00Intro
2. 1:23dat is an open source tool for sharing and collaborating on data
3. 4:36analogy time: lets talk about source control
4. 4:47life before git
5. 5:051. somehow get a zip of cool-project 2. unpack and edit a file 3. email the file back 4. ????
6. 5:53maintainer creates new zip of cool-project that might contain my fix
7. 7:05claim: currently data sharing is a mess
8. 7:30email csv files
9. 7:43we want to do for data what git did for source code
10. 7:56npm install -g dat
11. 10:27max, import your genome into dat
12. 11:47data is stored locally in leveldb blobs are stored in blob-stores
13. 13:07choose the blob store that fits your use case s3, local-fs
14. 13:32auto schema generation - free REST API - *all* APIs are streaming
15. 13:57a data set we can all relate
16. 17:02calculate how big npm is using dat
17. 18:43dat cat transform
18. 19:04dat cat docker run-i transform
19. 19:46transform the npm data using bulk-markdown-to-png
20. 24:56use case: trillian astronomical
21. 25:50problems: huge files, weird format
22. 26:131TB gzipped CSVS 600 million objects, 300 columns 40TB imagery
23. 31:59data pipelines dependency management data streaming
24. 33:36gasket is a cross platform pipeline manager
25. 35:00datscript is an experimental pipeline config language
26. 35:48the future
27. 36:04branches, dat checkout 3b2d98V3, multi master replication, sync to databases, registry

" related talks

• 22mDatMax Ogden · NodeConf ONE-SHOT Oakland 2014
• 25mManaging changes to your data with datKarissa McKelvey · EmpireNode 2015
• 29mData Stored Everywhere and Nowhere - Building Applications on the Distributed WebKarissa McKelvey · NodeConf Argentina 2017
• 13mLessons learned in productionMatt Eernisse · NodeConf 2011
• 24mA Real Database Rethink, Introducing LevelDB and NodeJSRod Vagg · NodeConf.eu 2013
• 33mTaking Node.js to The Ex-Stream: Stream to ScaleGadi Raymond · Node.js Global Summit 2020
• 36mData Versioning - does it mean what we think it means?Einat Orr · Data Con LA 2020
• 33mData.jsMichael Aufreiter · JSConf EU 2011