Data engineering in 10 years
summary
Predicts that data engineering industrializes into data factory engineering, moving from Hadoop and data lakes through the modern data stack and lakehouse to automated pipelines and infrastructure-as-code on Kubernetes and Spark. Gives concrete evidence: Spotify builds Discover Weekly with three engineers in three weeks and reaches 40 million active users in a year, mature shops produce millions of datasets per day while Google counts billions. Argues SQL and low‑code are interim tools that hit expressiveness and quality ceilings and that automated, testable production pipelines and reusable generation code actually work.
AI-written summary · about
chapters (26)
- 0:00Intro
- 0:37Prediction of future?
- 2:18How to predict the future?
- 4:42Vintage digital disruption - MRP
- 9:16New rules?
- 10:12What is the power of data engineering?
- 11:18Efficiency gap, data cost & value
- 12:47Data agility
- 13:41Enabling innovation
- 15:13Manual, mechanised, industrialised
- 18:17Crafted artifacts: data models
- 20:14Artisanal vs industrialised data modelling
- 22:00Premature modelling is waste
- 22:44Artisanal vs industrialised knowledge graphs
- 23:52Artisanal vs industrialised machine leaming models
- 24:29Road towards industrialisation
- 25:30Simplifying use of new technology
- 27:41Data engineering in the future
- 28:17Future of low-code & no-code
- 29:48SQL for data processing
- 32:17SQL inadequate for mature applications
- 34:21Technology adoption & modern data stack
- 35:29Technology adoption & data lake collaboration
- 35:52Data products / contracts = old rules, new context
- 36:49Extreme programming
- 37:06Agile
related talks
41mData engineering without walls
44mThe Art of Data Engineering
53mThe State of Data Engineering
30mDevOps For Data Teams
25mWhy Software Engineering Best Practices Fail in Data Engineering
42mTales from the Field: The Future is Now
42mWeb Development To Big Data. A Journey
19mDemystifying Modern Data Pipeline Architecture