Multi-statement Transactions in the Databricks Lakehouse
summary
Databricks adds multi-statement, multi-table transactions to the lakehouse by moving commits through catalog-managed commits, staging unpublished commits and then squashing them into a single atomic commit, using cloud put-if-absent for backing files and a glass pane in the engine to isolate transaction viewpoints. Spark, serverless gateways and notebooks all get changes; special work makes transactional copy into checkpoints safe under concurrency. The feature ships and works but forces cross-team plumbing, a 24‑hour/10‑minute lifetime limit and careful footprint and session restraints.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:27Delta Lake foundations
- 4:39Limitations of Delta Lake
- 10:50Catalog managed transactions
- 13:42Implementing multi-table support
- 18:38SQL and API challenges
- 25:38Engine and ingestion challenges
- 31:33Serverless transactional state
- 34:20Notebook and UX challenges
- 38:37Learnings and future work
- 42:26Q&A and closing
related talks
28mLakehouse Architecture with Delta Lake and Databricks
47mBeing transactional in 2019 - what does this actually mean?
51mSolving the challenges of adding transactional support to Data Lakes
45mBuilding a Lakehouse on the Microsoft Intelligent Data Platform
42mBuilding a Lakehouse data platform using Delta Lake, PySpark, and Trino
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
31mSimplifying upserts and deletes on Delta Lake tables
58mDon’t Drop ACID - Transactions in Distributed NoSQL