Bringing Software Engineering Rigour to Data, DataEngBytes 2022
summary
Argues that applying DORA metrics to pipelines reveals how slow data teams are (elite teams deploy > once/day) and cites startling gaps like 973x more frequent deployments and 6570x faster lead/restore times for elite software teams. Recommends five fixes: decentralize via data-mesh, build a self-serve templates platform, automate CI/CD and testing beyond simple quality checks, add observability and detective data-security, and stop tool sprawl (mentions Snowflake, Databricks, DBT, Redshift) so pipelines are both fast and reliable.
AI-written summary · about
chapters (28)
- 0:00Intro
- 2:28Welcome
- 3:27Software Application Development
- 4:36Data
- 5:42Testing
- 7:18DevOps Metrics
- 11:41Accelerated State of DevOps
- 12:38How do we become an elite data team
- 13:07Five failure points
- 14:02Slow moving data platforms
- 16:04Decentralizing data engineering
- 16:20Data Mesh
- 17:59Enabling Domains
- 19:14Domains on Platforms
- 20:08Combine Capabilities
- 20:47Automated Platforms
- 20:59Slow Platform Adoption
- 21:27Automated Domain Capabilities
- 23:02Overprotective
- 24:16What happened
- 25:16Detective approach
- 26:23Cluster of mass
- 26:33A true story
- 29:51The good things take time
- 30:11Start small
- 32:35References
- 33:15Sponsor Desk
- 33:34Distributed Data Mesh
related talks
38mKeeping pace with data infrastructure so it doesn't outrun you
42mData-Driven DevOps
30mDevOps For Data Teams
51mFrom DevOps for data to DataOps
27mData team strategies: unleashing your team
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
23mBridging the DevOps Gap in Modern Data Teams
22mDORA Metrics: The Key to DevOps Excellence