Simplifying upserts and deletes on Delta Lake tables
summary
Delta Lake lets you run atomic MERGE with UPDATE/INSERT/DELETE using SQL or Python/Scala and shows demo runs that report affected row counts and restore prior states with time travel via the _delta_log JSON transaction log. Behind the scenes merges use file min/max stats to limit reads, optimize compacts many small files into ~1 GB files and vacuum removes old files, and schema enforcement or optional schema evolution prevents or accepts new columns. Databricks reports 5,000+ customers and the demos show the operations succeed and produce predictable, auditable table versions.
AI-written summary · about
chapters (10)
- 0:00Introduction and agenda
- 1:30About Databricks and Delta Lake
- 3:42Challenges with traditional data lakes
- 10:03How Delta Lake solves these challenges
- 16:03Updates, deletes and upserts in Delta
- 19:54Optimize and vacuum in Delta
- 20:39Demo: Update and delete operations
- 24:12Demo: Schema enforcement and evolution
- 27:45Demo: Time travel and optimization
- 29:29Q&A and future of Delta Lake
related talks
45mSimplify your ETL with Databricks Delta Lake
20mDelta Lake: Turn your Data Lake into a Lakehouse
38mGetting Started with Delta Lake
58mBuild reliable data lakes with Delta Lake & Databricks
28mLakehouse Architecture with Delta Lake and Databricks
39mDelta Lake: Diving into Data Lakes Without the Downsides
01h 21mTransaction log as the heart of Delta Lake
21mBuilding Data Integration Services In Real Time