How Not to Build a Data Platform Lessons from Failure to Success
summary
Built a massive platform with IaC, Azure Kubernetes nodes, Data Factory pipelines, Synapse and hosted SQL plus dbt but it only served one file for two daily users so costs outrun value and the project is canceled. A 60 GB ingest and 200 million rows example proves scale is not the point when adoption is zero. Focus on high‑impact use cases, engage business stakeholders early, start infra small, train users and measure real value before scaling.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:09Data platform tools overview
- 3:37Over-engineered platform
- 5:24Business rejection and costs
- 7:12Lessons from failure
- 9:19High-impact use cases
- 10:41Stakeholder engagement
- 13:50Tool choice vs business needs
- 16:19Pain of failed builds
- 18:26Success story and path
- 19:31Stakeholder sessions
- 20:34Q&A
related talks
19mHow to Architect your Data Platform for Success
44mData Mesh: Lessons from the trenches
35mBuilding a data analytics platform on strong opinions held loosely
29mHow I built an entire data platform by myself that thousands of data engineers use in one year
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
07mFrom Scaling APIs to Scaling Data to Scaling Your Team
35mCloud Native Data Platforms - The why, what and how
26mEffective Data Migration: Challenges & Success