Introducing Data Mesh A Paradigm Shift In Analytical Data Management Architecture
summary
Argues that centralized data warehouses and lakes hit scale limits and proposes domain-owned, product-minded datasets connected by a platform that provides discoverability, lineage, SLOs and SDKs. Implements a data product model with polyglot input/output ports, append-only read APIs and point-in-time queries; relies on Kafka, blob storage and cloud warehouses like BigQuery/Snowflake but warns current clouds force brittle account and storage workarounds. Recommends platform teams plus domain data teams, federated governance encoded in the platform, and prototypes showing feasibility but requiring new tooling.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 1:08What is Data
- 3:27Data Warehouse
- 4:14Snowflake
- 4:55Model Data Lake
- 6:45GCP Cloud Architecture
- 7:27Characteristics
- 8:34Centralized monolithic architecture
- 9:38The job of an architect
- 10:52Execution
- 11:15Hexagon
- 11:35Scale
- 12:25Failure Modes
- 12:58Decomposition
- 16:40Empower
- 18:18Governance
- 20:47Pillars of Data Mesh
- 21:38Deep Dive
- 25:13The Mesh
- 27:32Data Infrastructure
- 28:35Team Structure
- 30:08Data Mesh Governance
- 32:27Whats Next
- 33:38Paradigm Shift
- 34:57Data Mesh Innovation
- 35:19Operational and Analytical Worlds
- 35:37Journey of a thousand months
related talks
48mData Mesh Paradigm Shift in Data Platform Architecture -
47mModern Data Architecture: Breaking the Big Data Monolith
39mData Mesh: Why, What and How
20mEmbracing Data Mesh: A Modern Strategy for Scalable Data Management
43mUntangling Data Mesh: A paradigm shift in Big Data
01h 00mData Mesh: Beyond the Data Lake
39mKeynote - Data Mesh
52mFrom Data Mess to Data Mesh: Navigating People, Process & Platforms