Building a data mesh
summary
Springer Nature moves from siloed reports to a domain‑owned data mesh built on BigQuery and DBT, with a thin self‑serve platform (Terraform, Cloud Run, catalog UI) and pipelines that push read‑only source data through raw → staging → presentation tables. Data is treated as a product with source‑aligned, consumer‑aligned and aggregated types, lineage and contract tests, masked columns for privacy, and federated governance. First product takes 14 months then 8 then integrates in 2 weeks, vendor exports and team incentives remain the hardest problems.
AI-written summary · about
chapters (38)
- 0:00Introduction
- 0:30About Charles
- 1:22About Springer Nature
- 3:25Challenges with data
- 4:49Opportunities
- 5:09What do we need
- 6:08Data mesh
- 6:31Data accessibility
- 7:29Data types
- 8:34Data pipelines
- 10:51Treating data as a product
- 12:10Domain orientation
- 12:32Bounded context
- 13:24Selfserve data platform
- 15:10Federated computational governance
- 16:22Data products
- 17:15Source aligned
- 18:04Data pipeline stages
- 20:05Consumer Line
- 21:49Consumer Pipeline
- 22:51Aggregated Pipeline
- 25:28Use cases
- 26:32How it went
- 27:04Finding the data owners
- 27:55Working with the domain team
- 29:13Data analysis
- 30:21Technology
- 31:53DBT
- 32:59Lessons learned
- 33:49Use cases for data
- 34:42Incentive
- 36:18Source systems
- 37:09Data engineering
- 37:19DBT and SQL
- 38:02Multidisciplinary teams
- 38:58Data quality
- 41:18AI
- 43:04Questions
related talks
38mData Mesh, Now
39mData Mesh: Why, What and How
23mHow To “Data Mesh”: What You Need To Know To Make It Real
44mData Mesh
36mIntroducing Data Mesh A Paradigm Shift In Analytical Data Management Architecture
20mEmbracing Data Mesh: A Modern Strategy for Scalable Data Management
31mData Mesh In Practice – How To Set Up A Data Driven Organization
43mUntangling Data Mesh: A paradigm shift in Big Data