Tales of Data Architecture Evolution
summary
Starts with a single relational DB and a one-off script that becomes a brittle ETL job then scales to a data lake on cheap blob storage (S3/HDFS) holding ~10 years or petabytes of raw events, orchestrated by Apache Airflow, feeding a modern cloud warehouse (Snowflake/BigQuery) and columnar formats (Parquet/Iceberg/Delta). Next shift moves ETL to streaming with Kafka and CDC for seconds‑level latency, then platform automation to deliver self‑service data products and finally organization‑level governance toward a data mesh.
AI-written summary · about
chapters (12)
- 0:00Introduction and welcome
- 1:11What is data architecture
- 3:19Audience and speaker introduction
- 7:12Day one data architecture
- 13:32From scripts to data warehouse
- 15:18Evolution to data lake
- 18:28Problems with data lake
- 21:37Real-time streaming architecture
- 27:11Infrastructure automation
- 28:37Democratizing data
- 31:46Modern data platform complexities
- 32:50Sustainable architecture and next steps
related talks
41mOn Data: Sky above, sand below, peace within
41mModern Data Architectures – What They Are and How to Implement?
52mData Warehouse, Data Lake, Data Mesh, Oh My: The History of Data Storage
42mWeb Development To Big Data. A Journey
45mModern Data Architecture
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
30mData Engineering building your BI infrastructure from scratch
44mThe Changing Face of ETL: Event-Driven Architectures for Data Engineers