A Guide to the Modern Data Stack
summary
Claims that moving from on‑prem ETL to a cloud ELT not ETL pipeline cuts delivery from months or years to days or hours, offloads infra to cloud, and makes components plug-and-play cloud services. Concrete toolset: ingest with Fivetran/Stitch or Spark/Kafka, store on BigQuery/Snowflake/Databricks, transform with dbt, orchestrate with Airflow and push decisions back via reverse ETL. Live demo connects dbt to BigQuery, builds source→models→fact with lineage and tests and runs successfully, showing versioned SQL and docs — dbt lineage and tests.
AI-written summary · about
chapters (11)
- 0:00Modern data stack intro
- 4:42Traditional stacks challenges
- 9:30Need for modern stack
- 25:27Modern stack characteristics
- 28:22Core layers of stack
- 36:11Data quality with observability
- 40:08Orchestration and reverse ETL
- 45:20Choosing tools per layer
- 1:04:23Benefits of modern stack
- 1:15:19DBT demo
- 1:26:39Q&A
related talks
16mETL vs ELT: Why the T has shifted
38mReigniting the ETL vs. ELT Debate - Which Better Supports the Business?
01h 35mThe Modern Data Stack
41mModern Data Architectures – What They Are and How to Implement?
39mLeaving ETLs Behind: Moving from a ‘Code Less’ Solution to a ‘Code Full’ Approach
26mETL pipeline with GCP Dataflow
38mThe Rise & Downfall of the Data Engineer REVISITED
50mMasterclass: Building a data stack in 2021