Why Software Engineering Best Practices Fail in Data Engineering
summary
Empirical claim: analysis of 400 million pipeline runs finds a 7% prod failure floor and shows deploys don't cause failures — 24‑hour before/after deploy rates are flat. Failures concentrate: 3% of DAGs cause 67% of failures, plus buggy runtimes, long durations, schema drift and late data. The useful content maps to a five‑layer framework and four concrete fixes: contract tests for inputs, time monitoring, state‑safe publishes, and graceful degradation.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:30Deploys cause failures?
- 3:32Testing deploy hypothesis
- 7:24Software assumptions fail
- 10:14Inversion app vs data eng
- 13:41CI/CD misses most failures
- 17:34Five-layer reliability framework
- 21:45Four moves to improve reliability
- 23:54Summary
related talks
53mThe State of Data Engineering
31mWhy Most Data Projects Fail and How to Avoid It
44mThe Art of Data Engineering
32mWhy Most Data Projects Fail and How to Avoid It
37mEngineering data quality
16mBeyond ETL: Revolutionizing Data Engineering Workflows
41mData engineering in 10 years
40mThe Hidden Cost of Data Debt