What Data Engineering Can Learn from Front-end Engineering
summary
Cloud warehouses (BigQuery 2010, Redshift 2012, Snowflake 2014) enable more ambitious pipelines, producing thousand-line SQL and brittle imperative orchestration; dbt and webpack-style compilation solve modularity but add build/debug overhead. Expect a shift to declarative reconciliation (orchestrators that diff desired vs actual data assets), wider adoption of static typing (TypeScript-like checkers for Python and a TypeScript for SQL), per-branch preview deployments (Vercel, Snowflake zero-copy clones) and consolidation around a few toolchains.
AI-written summary · about
chapters (8)
- 0:00Start
- 3:12The platform shift
- 6:14Two problems with the JS stack 10
- 7:36Data went through a similar transition.
- 10:25Real Engineering
- 17:37Life without an IDE sucks.
- 19:25What can data learn from frontend?
- 30:27Templating → Programming languages.
related talks
38mThe Rise & Downfall of the Data Engineer REVISITED
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
31mFuture of data engineering: what's in store?
40mData Engineering in a Rapidly Changing World
41mModern Data Architectures – What They Are and How to Implement?
39mWhat if SQL was back in fashion?
16mETL vs ELT: Why the T has shifted
51mPanel Debate: Rise Of The Data Engineer