Panel: Open Data Lakes

on YouTube ↗31m

" summary

Panel lays out a concrete PHAS3 stack: Presto for interactive SQL, Apache Hudi for atomic writes, AWS Glue as a serverless metadata catalog, and S3 for storage. Hudi adds transaction layer features — atomic file commits, updates/deletes, CDC, MVCC and optimistic concurrency — powering production lakes at TikTok (exabyte) and Uber (250–300 PB). Engines consume open formats like parquet with bloom-filter and other optimizations, enabling snapshot queries/time travel, multi-engine compatibility, and easier migration without vendor lock-in.

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 2:57Vinoth Chandar
3. 7:23Dipti Borkar | Ahana
4. 15:28Roy Hasson
5. 26:13Eric Kavanagh

" related talks

• 25mBuild open format data lakehouseSatish Mane, Rajeev Jahswal · Conf42 Cloud Native 2023
• 31mBuilding large scale, transactional data lakes with Apache HudiNishith Agarwal · Big Data & AI Toronto 2020
• 33mThe advent of the open data lakeJulien Le Dem · AI By the Bay 2025
• 50mApache Hudi : The Data Lake PlatformVinoth Chandar · ApacheCon@Home 2021
• 24mData Mesh and LakehouseMatei Zaharia · State of Data Mesh 2022
• 22mPresto for Your Fast, Open SQL LakehouseSteven Mih · OSA Con 2022
• 40mUnbundling of the Cloud Data Warehouse: Open Source Databases and Data LakesZoe Steinkamp · Jfokus 2026
• 51mPresto Open Source SQL Engine for Massive Scale Data LakehousesJames Cho · Open AI + Data Forum @ Open Source Summit North America 2023