Gone Streaming: dbt+Materialize
summary
Claim: dbt cannot keep models fresh for streaming data but paired with Materialize it does. The setup uses the dbt-materialize adapter to create incremental materialized views in Materialize backed by differential/timely dataflow. A demo ingests PubNub market orders, parses symbol and bid_price, computes per-symbol running averages, runs dbt once and shows live millisecond-updates via SELECT queries. Result is continuous correct averages without repeated dbt runs, with minimal recomputation as new rows arrive.
AI-written summary · about
chapters (10)
- 0:00Start
- 2:49How does dbt work?
- 4:26dbt models
- 6:04"materialization" determines how the transformation is persisted
- 7:35Materializations and data updates
- 8:25Batch data versus streaming data
- 9:10dbt and batch data
- 10:25dbt and streaming data
- 12:03How does Materialize work?
- 13:39Incremental materialized views
related talks
49mdbt: A Software Engineering Approach To The Analytics Workflow
23mDelivering Modern Data Stacks With dbt to Operationalise Analytics
01h 42mFall in Love with Data Transformation: DBT as Your Weapon
52mMaking sense of Crypto Market Data in real time
05mDBTB INT Shinji Kim
24mAsset based orchestration: the future of data orchestration
03mDBTB INT Monal Daxini
01h 07mHow to Count Things with dbt - Data Build Tool