Building large scale, transactional data lakes with Apache Hudi
summary
Apache Hudi implements ACID transactional lakes on DFS using Avro/Parquet, Spark/Hive/Presto clients and Kafka ingestion, and runs in production at Uber at 150+ TB, managing tens of thousands of tables and ingesting ~500 billion records/day. It provides atomic multi-row commits, snapshot isolation, delta streamer for 2–5 minute freshness, time travel and restore points, bootstrap conversion of existing tables, and adaptive layout/compaction plans. The stack is engine-agnostic in progress and is proven to support incremental ETL, feature stores and large-scale deletes/updates.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:26Incremental Database Ingestion?
- 2:20De-Duping Log Events?
- 3:36#R3: Storage Management?
- 4:32Scaling DFS/Storage RPCS?
- 5:21#RS: Transactional Writes?
- 6:24Faster Derived/ETL Data
- 7:27Compliance/ Data Deletions?
- 8:49Data Lake - Data platform
- 9:37#RI: Managed Ingestion
- 10:39#R2: Unify Stream & Batch?
- 12:48Introducing Apache Hudi
- 14:10Hudi-Based Data Lake Untled analytical storage
- 15:33#Hudi Guarantees
- 16:22#Data Platform Ecosystem
- 18:02HoodieDeltaStreamer Out of the box solutions
- 19:11Spark Datasource integration out of the box solutions
- 20:02Recover from data corruption Restoring tables
- 21:18Incremental ETL
- 22:04Time Travel queries
- 22:49Easy conversion of existing table into Huditable
- 23:45Roadmap
related talks
50mApache Hudi : The Data Lake Platform
01h 08mApache Hudi: A Database Layer Over Cloud Storage for Fast Mutations & Queries
31mPanel: Open Data Lakes
25mBuild open format data lakehouse
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
25mDelta Lake has competition
41mTransactional data lakes for your analytical workloads
51mSolving the challenges of adding transactional support to Data Lakes