A Git-like Repository for Your Data Lake
summary
A Git-like layer sits on top of object stores and presents S3/GCS/any object store as a versioned data repo where branches and commits point at objects rather than duplicating them. The demo shows a 2 TB branch copy created instantly (no full data copy), parquet review files tracked, and integration with Airflow, Kafka and tools that write data. Operations run via CLI, HTTP, SDKs; commits are atomic across many files, support pre-commit checks, metadata links back to notebooks, and the demo works end-to-end.
AI-written summary · about
related talks
26mCI/CD For Data – Building Data Development Environments With Open Source Stacks
31mGetting ML Right in a Complex Data World
38mML Data Version Control and Reproducibility at Scale
20mCI/CD for Data - Building Dev/Test Data Environments with Open Source Stacks
22mBig data tools
22mData Exploration Made Easy with Open Source
20mDelta Lake: Turn your Data Lake into a Lakehouse
11mIntroducing Dat: If Git Were Designed For Big Data