From Warehouse to Lakehouse: Data at Rest
summary
Starts from MySQL + Redshift, moves to S3 + Kafka ingestion, then replaces AWS firehose/Kinesis with self‑managed Flink and Kafka and standardizes on Iceberg tables with Nessie catalog, Trino for ad‑hoc SQL and Spark for ETL. Claims cut infrastructure cost (about 90% open source stack) and restores updates/deletes, schema evolution and time travel with snapshot IDs, plus Git‑like branching and merges via Nessie so migrations and fixes run on a branch then merge into main. System runs in Kubernetes with GitOps and the lakehouse setup works in production.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 6:46Starting with warehouse and monolith
- 11:40Challenges with data lake architecture
- 14:28Introducing lakehouse and table formats
- 19:16Schema management and ingestion
- 23:55Reading and reprocessing data
- 27:09Data versioning and time travel
- 29:36Final architecture and benefits
related talks
37mBuild a Modern Data Platform on AWS
28mStreaming Lakehouse with Kafka, Flink and Iceberg
34mCatalogs, Iceberg, and Delta: Building the Foundation for Lakehouse Analytics
26mArchitecting real-time transactional data lake
25mBuild open format data lakehouse
41mOn Data: Sky above, sand below, peace within
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
29mData Acquisition: Overcoming Challenges