BACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
summary
Seven concrete trends claim to reshape data work: prioritize start with lineage — build data lineage and discovery first when creating a new warehouse; expect data quality to cost economies (Microsoft cites $3.1 trillion) so add observability and anomaly detection; adopt dbt + Great Expectations with orchestration like Airflow Dagster or Prefect; evaluate lakehouse engines (Iceberg/Delta/Hudi) for incremental SQL on object storage; weigh domain-driven data mesh and MDM tradeoffs; industrialize ML with TensorFlow PyTorch transformers and TPUs; treat privacy, bias and reverse‑ETL as operational problems.
AI-written summary · about
chapters (10)
- 0:00Start
- 7:43Data Discovery & Metadata Management
- 12:11Data Mesh & Domain Ownership
- 17:22Catalog The Mes(h)s
- 18:13Data Observability
- 23:17Data LakeHouse
- 27:00Modern Data Stack
- 32:54Industrialized ML
- 36:36Diversity, Privacy, Al Ethics
- 57:23craft hub
related talks
31mFuture of data engineering: what's in store?
40mData Engineering in a Rapidly Changing World
38mThe Rise & Downfall of the Data Engineer REVISITED
26mTrends Shaping The Next Decade In Data
39mWhat Data Engineering Can Learn from Front-end Engineering
11mRecent trends in data and machine learning technologies
41mModern Data Architectures – What They Are and How to Implement?
51mPanel Debate: Rise Of The Data Engineer