Future of data engineering: what's in store?
summary
Panel predicts tight budgets force teams to do more with less, driving investment in orchestration and unification tools to replace manual stitching. Specific tech calls include Snowpark + dbt collapsing the SQL/Python divide, merging warehouse/lake patterns, and rising interest in data mesh and metric layers. Teams measure impact with per cost per conversion and favor buying commodity tooling while building strategic differentiators. Monte Carlo is used to give marketing analysts self-serve alerts and task lists, improving detection and local remediation of data issues.
AI-written summary · about
chapters (10)
- 0:00Introduction and speakers
- 2:342023 data team trends
- 8:10Data platform scaling strategies
- 11:42Self-serve data platforms
- 17:14Build vs buy tools
- 19:44Centralized vs decentralized teams
- 22:14Data reliability and ownership
- 26:31Data observability outcomes
- 28:00Data contracts and experimentation
- 30:50Closing remarks
related talks
50mThe future of Data Engineering panel
51mPanel Debate: Rise Of The Data Engineer
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
38mThe Rise & Downfall of the Data Engineer REVISITED
40mData Engineering in a Rapidly Changing World
39mWhat Data Engineering Can Learn from Front-end Engineering
23mData Team as an Optimisation Problem
29mBuilding successful data teams in the realities of 2023