Data Mesh Paradigm Shift in Data Platform Architecture -
summary
Centralized data lakes and warehouses are the bottleneck: Fortune 1000 firms increase big data and AI spend while confidence in outcomes falls, and data scientists spend ~80% of their time cleaning. Data Mesh claims a working alternative: partition by domain data sets (source-oriented and consumer-oriented), deliver them as data products, back them with a thin self-serve platform (cloud storage, Spark/Hadoop, Airflow, GCP services, unified access control) and federated governance. Success measures: discoverability and lead time from find to consumption; quick scaffolds often reduce setup to minutes or hours.
AI-written summary · about
chapters (16)
- 0:00Introduction and speaker background
- 1:01What is a paradigm shift
- 3:58Crisis in data architecture today
- 6:11Current data architecture divide
- 7:42Generations of data approaches
- 13:25Why centralized solutions fail
- 17:21Separation by technology issues
- 24:54Stuck in normal science phase
- 25:15Introducing data mesh concepts
- 26:28Domain-driven distributed architecture
- 31:13Product thinking for data sets
- 37:18Platform thinking and infrastructure
- 41:12Ecosystem governance standards
- 43:40Distributed ownership model
- 45:47New language for data mesh
- 47:13Closing and further resources
related talks
36mIntroducing Data Mesh A Paradigm Shift In Analytical Data Management Architecture
47mModern Data Architecture: Breaking the Big Data Monolith
43mUntangling Data Mesh: A paradigm shift in Big Data
39mData Mesh: Why, What and How
01h 00mData Mesh: Beyond the Data Lake
20mEmbracing Data Mesh: A Modern Strategy for Scalable Data Management
50mData Mesh: the new paradigm for data management
39mKeynote - Data Mesh