Next Gen ETL Generators
summary
Argues that ELT generators remain practical for cloud warehouses by parsing an Excel spec or SQL to auto-produce dbt models, Matilion JSON jobs, Snowflake/BigQuery stored-procedure templates and standard staging loads (including SCD type 1/2 patterns). Claims this cuts development time and standardizes pipelines, reports successful on multiple projects, but notes portability between runtimes is theoretical and maintenance requires strict single-source editing of the spec; deployment follows normal CI/CD after generation.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:43Data architecture evolution
- 3:11ETL tool generations
- 4:14ELT generators defined
- 6:19Excel for warehouse specs
- 7:22Automate ETL from SQL
- 8:04Code for Matillion & dbt
- 9:10Staging & procedures
- 10:12Generators for BigQuery
- 11:17Microsoft legacy integrations
- 11:38ELT automation benefits
- 13:26Conclusion & Q&A
related talks
16mETL vs ELT: Why the T has shifted
38mReigniting the ETL vs. ELT Debate - Which Better Supports the Business?
01h 28mA Guide to the Modern Data Stack
55mThe ELT pattern in action with DBT and SQLMesh
39mWhat Data Engineering Can Learn from Front-end Engineering
39mLeaving ETLs Behind: Moving from a ‘Code Less’ Solution to a ‘Code Full’ Approach
39mWhat if SQL was back in fashion?
31mMetadata-based Dynamic ETLs