Data Exploration Made Easy with Open Source
summary
Claims that adding embedded DuckDB to the lakeFS UI lets users run real SQL directly on Parquet/CSV in object stores with low latency — an example shows a select limit 10 drop from ~7 seconds to 159 ms. lakeFS overlays git-like branches and commit metadata on S3 without copying petabytes, so users compare versions, revert commits and inspect who changed what. Project arises from a real incident managing 8.5 PB where a bug deleted ~1 PB (750 million objects) and drove the design.
AI-written summary · about
chapters (12)
- 0:00Introduction to data exploration
- 1:27Defining data exploration
- 3:55Challenges with traditional tools
- 7:25Time dimension in data exploration
- 8:27Space dimension in data exploration
- 9:54Human behavior in data
- 10:15Solutions for data exploration
- 10:36Introducing DuckDB
- 14:35Introducing LakeFS
- 17:03Using branches for data changes
- 19:35Open source tools availability
- 20:29Q&A and real-world use
related talks
55mDuckLake: Learning from Cloud Data Warehouses to Build a Robust “Lakehouse”
26mBuilding Data Lake Using DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
22mBig data tools
39mDelta Lake: Diving into Data Lakes Without the Downsides
31mGetting ML Right in a Complex Data World
45mAnalytics for not so big data with DuckDB
20mDelta Lake: Turn your Data Lake into a Lakehouse