Beyond dbt: Modern SQL Transformation and Lineage with SQLGlot and SQLMesh
summary
SQLGlot is shown as a 24‑dialect parser built on an AST with a Rust tokenizer and reproducible dialect‑agnostic transpilation, enabling column‑level lineage extraction that handles CTEs and CASE logic after some tweaks; SQLMesh provides state logging for cheap incremental runs plus time‑travel, UI‑driven backfills and virtual/physical model layering; a DIY pipeline is built parsing compiled SQL to a DB and it mostly works but needs schema injection to resolve select ; market size quoted is $1.52 billion*.
AI-written summary · about
chapters (26)
- 0:00Introduction
- 1:47Data lineage market overview
- 3:11What is dbt
- 4:17SQLGlot engine
- 6:21Abstract syntax trees
- 7:25SQLMesh components
- 8:07SQLMesh UI workflow
- 9:29SQLMesh vs dbt
- 12:41SQLMesh model testing
- 13:46Breaking changes handling
- 15:35Physical and virtual layers
- 16:38Lineage tools comparison
- 18:26Open Lineage limits
- 20:31Column level lineage
- 23:01Handling SELECT *
- 24:05SQLGlot with dbt
- 25:11Custom lineage challenges
- 26:40Q&A
- 27:45Fiverr acquisition impact
- 29:10Monolithic lock-in risks
- 30:34Enterprise SQLMesh use
- 31:19SQLMesh and dbt improvements
- 32:23SQLMesh performance
- 33:06LLMs for SQL parsing
- 34:09Vendor flexibility
- 35:12SQLMesh in data lifecycle
related talks
55mThe ELT pattern in action with DBT and SQLMesh
50mModern SQL: Evolution of a dinosaur
49mdbt: A Software Engineering Approach To The Analytics Workflow
45mMore Than a Query Language: SQL in the 21st Century
30mThe Future of Data is Words
32mBecause SQL is everywhere but you don't want to use it
47mModern SQL: Evolution of a Dinosaur
34mColumn-level lineage is coming to the rescue