Tales of Data Architecture Evolution
summary
Starts with a single operational database and a brittle script that becomes the first ad hoc data warehouse, then evolves to an ETL-driven Airflow and S3 lakes holding petabytes and 10 years of raw data for analysts; next shift adds streaming with Kafka with CDC and Debezium to cut latency, enable real‑time joins against historical lakes, and stream into Snowflake/Spark/Trino for analytics. Expect problems: pipelines taking 10–12 hours, out‑of‑sync deletes, exploding asset counts, governance headaches and GDPR fines, plus rising use of embeddings and vector DBs for AI.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:43Data Architecture
- 3:05Evolution
- 5:33System Architecture
- 10:11Microservices
- 13:30The Next Paradigm Shift
- 15:51Data Lake
- 22:18Data Loss
- 25:43Data Latency
- 26:59RealTime Streaming
- 34:01Lambda Architecture
- 38:49Optimize
- 39:55Big enough
- 42:23Modern view
- 45:29Data without information
- 46:04Data management governance
- 48:06Data Mesh
- 49:35The Future
related talks
30mData Engineering building your BI infrastructure from scratch
41mModern Data Architectures – What They Are and How to Implement?
31mData Unleashed: A Developer's Perspective on Navigating the Architecture Maze
56mData Unleashed: A Developer's Perspective on Navigating the Architecture Maze
11mBreaking Data Silos: Enterprise Data Management
18mOur Analytical Data Journey – from Paper to Petabytes
25mPlatform Engineering for Modern Data Infrastructure
27mThe Evolution of Data Architecture