Workflows of Highly Functional App & Data Engineering Teams
summary
Outlines a pragmatic medallion pipeline: raw bronze dumps (hourly/daily) into a shared lake, cleaned and joined into silver, then precomputed, business-aligned datasets in gold; warns about nightly latency (you cannot beat the speed of light) and large duplicate storage costs being trivial in cloud. Recommends containerized Data API Builder with a JSON config to auto‑expose REST/GraphQL endpoints, advocates running databases in Docker, using database projects, and adding unit tests for stored procedures with an assert schema to catch regressions.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 1:08Why this talk matters
- 3:13Medallion strategy for data
- 8:31Bronze layer explained
- 10:56Silver layer transformation
- 11:36Gold layer insights
- 15:45Security and governance
- 17:10AI in development
- 19:58Real-time data trade-offs
- 23:02Data strategy philosophy
- 24:47Data quality issues
- 25:09ETL pitfalls
- 27:39Data mesh strategy
- 29:43Introducing data API Builder
- 31:29Cost of API coding
- 33:15Simplifying API development
- 34:40Demo of data API Builder
- 38:13Database regressions testing
- 40:43Stored procedure testing
- 41:46Database CI/CD integration
- 43:33TSQL linting
- 45:20Modern SQL tooling
- 46:43Strategic goals beyond silver
- 47:26Closing thoughts
- 48:08Q&A
related talks
01h 05mWorkflows of Highly Functional App & Data Engineering Teams
38mKeeping pace with data infrastructure so it doesn't outrun you
41mModern Data Architectures – What They Are and How to Implement?
27mData team strategies: unleashing your team
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
38mData Secrets from a Platform Engineer
30mDesigning Resilient Data Pipelines
23mBridging the DevOps Gap in Modern Data Teams