The Rise & Downfall of the Data Engineer REVISITED
summary
Revisits 2017 claims and argues the data stack now favors hosted pay-as-you-go platforms (Snowflake, BigQuery), shrinking internal infra teams while ELT in-warehouse transforms dominate; tools named include Apache Airflow, Apache Superset, dbt, Airbyte, Meltano and Preset powering pipelines; templated SQL and dbt macros proliferate and spawn mountains of templated SQL; analysts become analytics engineers (SQL+git); priorities shift to data ops and observability, decentralization (data mesh) and computation frameworks for repeatable metrics.
AI-written summary · about
chapters (16)
- 0:00Introduction and keynote setup
- 1:05Career and open source journey
- 3:12Purpose and scope of the talk
- 6:21Pay-as-you-go data infrastructure
- 9:53Rise of data integration services
- 13:23ELT over ETL trend
- 16:10Mountains of templated SQL
- 19:04Rise of the analytics engineer
- 22:34Data literacy and specialization
- 24:00Democratization of analytics
- 26:49Every product as a data product
- 27:53Emergence of data observability
- 29:22Erosion of BI semantic layer
- 31:28Decentralization of ownership
- 33:13Computation frameworks evolution
- 36:24Conclusions and future opportunities
related talks
51mPanel Debate: Rise Of The Data Engineer
39mWhat Data Engineering Can Learn from Front-end Engineering
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
40mData Engineering in a Rapidly Changing World
31mFuture of data engineering: what's in store?
50mMasterclass: Building a data stack in 2021
39mWhat if SQL was back in fashion?
34mTales of Data Architecture Evolution