Data Mesh in Practice
summary
Zalando shifts from a centralized data lake to a domain-driven Data Mesh, moving ownership to producing teams while keeping central governance for interoperability. They store about 10–15 petabytes in AWS S3, ingest via a central event bus, run queries with Presto and Databricks/Spark, and let teams bring your own bucket to opt into archiving. Thousands of datasets reveal most are unused while a small set of “golden” datasets drive production and ML; self‑service templates and cluster provisioning cut central bottlenecks and improve quality.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:52Who are we?
- 7:54Legacy Analytics
- 9:12Legacy Evolving
- 10:43Zalando's Data Lake
- 15:02Centralization Challenges
- 18:15A Recurring Pattern
- 23:40What is Data Mesh?
- 24:44Data as a Product
- 26:30Domain-Driven Distributed Data Architecture
- 31:20Self-Service Data Infrastructure
- 33:36An Ecosystem of Data Products
- 34:31It's a mindset shift
- 36:31Data Mesh in Practice
- 37:37Central Services with Global Interoperability
- 39:04Bring Your Own Bucket (BYOB)
- 40:40Central Processing Platform
- 42:25Simplify Data Sharing
- 45:09How to Ensure Data Quality?
- 50:40Into the Future
related talks
37mMetadata Automation in the Data Mesh
39mData Mesh in the wild
40mData Mesh, Data Fabric and DataOps; what should you choose?
36mData Mesh and Its Superpowers
36mData Mesh: Kicking Off a Mentality Change in an Organisation
47mData Mesh in Kubernetes
46mA Data Mesh Reference Implementation on AWS
51mWhat is a Data mesh?