Data Engineering in a Rapidly Changing World
summary
Defines data engineering as managing the data engineering lifecycle: ingesting from systems, transforming, storing and serving for analytics or ML, not tool fetish. Points out cloud-era realities: Redshift made infra cheap ($0.25/hr), now costs spiral, so expect consolidation, rising importance of FinOps, and shifts toward managed stacks (Snowflake, Databricks, BigQuery). Highlights small-data tools (DuckDB / MotherDuck) and warns prototypes often become production. Advises mastering fundamentals (databases, streaming, lean practices) and focusing on people and process.
AI-written summary · about
chapters (9)
- 0:00Intro
- 2:29What is data engineering
- 5:07What is a data engineer
- 6:29History of data engineering
- 8:17Rapid change
- 13:23Career paths
- 16:49Staying relevant
- 24:36Focus on today
- 30:02Wrap up
related talks
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
25m"Data Engineering: The World Between Worlds"
38mThe Rise & Downfall of the Data Engineer REVISITED
51mPanel Debate: Rise Of The Data Engineer
22mDataEngBytes 2023 - BRI-T1-06 - Yanir Seroussi
24mGotta Feed the Beast - Data Engineering at Scale
31mFuture of data engineering: what's in store?
30mData Engineering building your BI infrastructure from scratch