Cloud Native Data Platforms - The why, what and how
summary
Argues that cloud native patterns are colliding with big data and that companies move from centralized warehouses to data mesh domain-owned data products using concrete tools like S3, Databricks, Airflow, dbt and Dexter. Describes hundreds of schema changes, platforms that fail when scaled from tens to 10,000 users, and the use of data contracts and software defined data assets to detect schema drift, run declarative pipelines in containers, and make distributed, self‑service platforms actually work.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:05Data platform consultancy
- 2:28Business dream vs data reality
- 4:15Data platforms history
- 8:10Why data lakes failed
- 10:02Data products and ownership
- 12:28Data needs context
- 14:58Data contracts
- 16:48Designing data for humans
- 18:36Cloud native and Data Mesh
- 20:00Modern stack evolution
- 21:46Declarative pipelines with Dexter
- 23:33Data platforms at scale
- 24:59Shift to cloud native
- 27:06Machine learning’s demand for better data
- 28:31Cloud native future
- 31:00Operational meets analytical
- 32:04Philosophy over tools
- 32:48Defining domains
- 34:00Q&A
related talks
32mCloud-Native Data: What is it? - Majors, Dunlap, O’Grady, Richardson, Drewitz
36mIntroducing Data Mesh A Paradigm Shift In Analytical Data Management Architecture
39mKeynote - Data Mesh
22mData Divide: End User’s Cloud Native Journey to Distributed Databases
47mModern Data Architecture: Breaking the Big Data Monolith
30mPlatform Oriented Data Architecture for Acceler...
39mData Mesh: Why, What and How
48mData Mesh Paradigm Shift in Data Platform Architecture -