Data Mesh: Challenges and Opportunities
summary
Panelists report concrete outcomes not slogans: teams break a monolithic warehouse into data products with automated platform checks, migrate on‑prem to cloud, and enforce federated computational governance so ownership, lineage and PII masking are automatic. Early wins include a first customer‑facing analytical product that restores trust, a claimed saving of "200,000k" in a year, and a central team growing from 0 to 150 in four years. Tools called out include Atlan, Airflow, Starburst and cloud storage; challenges are people, processes and tricky cross‑team modeling.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:55Panelists' backgrounds
- 4:22What is data mesh
- 7:54Data mesh for large corps
- 9:20Adapting to org size
- 12:33Signs company needs it
- 17:11Sustaining momentum
- 24:16First approach for adoption
- 29:04Biggest implementation challenges
- 34:24Cultural hurdles
- 40:08Data as product
- 45:09Standards across teams
- 48:32Automating data modeling
- 50:23Core systems and publishing
- 53:13AI-driven data access
- 57:07Environmental impact
- 58:52Closing thoughts
related talks
20mDataEngBytes 2023 - MEL-T1-10 - Josh Devlin
49mOne Year in a Data Mesh at PayPal
56mData Unleashed: A Developer's Perspective on Navigating the Architecture Maze
35mBuilding a data analytics platform on strong opinions held loosely
33mData: The Land DevOps Forgot
31mThe mesh inside the mesh
30mA Devops Approach to Data: A Tale Through the Trenches in Search of a Cloud Data Revolution
25mPlatform Engineering for Modern Data Infrastructure