Building Data Integration Services In Real Time
summary
Delta Lake on Databricks delivers ACID transactions and time travel to fix real failures: a demo pipeline merges Kafka streaming flight events, a slowly changing plane CSV, and five‑minute weather JSONs into silver/gold tables and shows fixes for half‑finished loads, incompatible schema, concurrent writers, accidental overwrites, and tiny‑file slowdown using compaction and data skipping. Atlassian’s Socrates uses Delta for GDPR: native DELETE/UPDATE instead of triple copying, implicit versioned backups, configurable VACUUM, serving ~2,000 monthly users and planning automated forget requests and Athena migrations.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:24databricks
- 1:18The Promise of the Data Lake
- 1:53Challenges of the Data Lake
- 3:59What is Delta Lake?
- 4:56Let's look at a use case
- 5:49Implementation Scenario: Before Delta
- 6:54Failed production jobs
- 7:53Lack of Schema enforcement
- 8:41Problem 3: Lack of Consistency
- 9:45Cannot recover lost data
- 11:52Partitioning is static
- 12:44GDPR compliance
- 15:31Personally identifiable Information
- 16:55The old way
- 20:31VIRTUAL EVENT
related talks
58mBuild reliable data lakes with Delta Lake & Databricks
45mSimplify your ETL with Databricks Delta Lake
22mReliable data engineering made easy
38mGetting Started with Delta Lake
45mThe Data Lakehouse: A tech talk for everyone who loves data
28mLakehouse Architecture with Delta Lake and Databricks
39mData Observability and Reliability Engineering in a Real-Time World
31mSimplifying upserts and deletes on Delta Lake tables