Apache Hudi : The Data Lake Platform

on YouTube ↗50m

" summary

Apache Hudi provides transactional updates/deletes and record-level incremental change streams over object storage, with deployed users at ByteDance and Uber handling hundreds of petabytes and exabytes. It supports Spark, Flink, Presto and offers two storage tradeoffs: copy-on-write (columnar reads) and merge-on-read (log+base for high-update workloads). Platform pieces include a metadata metaserver, planned caching layer, compaction/clustering daemons, and a Kafka Connect sync; multi-writer throughput currently targets roughly one commit per minute using optimistic concurrency control.

AI-written summary · about

" chapters (5)

1. 0:00Introduction and speaker background
2. 2:06Origins and motivation for Hudi
3. 13:51Project evolution and community contributions
4. 22:20Hudi as a data lake platform
5. 37:11Q&A session

" related talks

• 31mBuilding large scale, transactional data lakes with Apache HudiNishith Agarwal · Big Data & AI Toronto 2020
• 01h 08mApache Hudi: A Database Layer Over Cloud Storage for Fast Mutations & QueriesVinoth Chandar · Future Data Systems Seminar Series (Fall 2025)
• 31mPanel: Open Data LakesDipti Borkar, Vinoth Chandar, Roy Hasson, Eric Kavanagh · OSA Con 2021
• 25mDelta Lake has competitionAlban Phélip · DataXDay 2020
• 25mBuild open format data lakehouseSatish Mane, Rajeev Jahswal · Conf42 Cloud Native 2023
• 22mThe Who, What, and Why of Data Lake Table FormatsAndrew Madson · Commit Your Code 2024
• 36mData Warehouse, Data Lakes, What's Next?Xiaowei Jiang · Flink Forward Virtual 2020
• 40mUnbundling of the Cloud Data Warehouse: Open Source Databases and Data LakesZoe Steinkamp · Jfokus 2026