Data pipelines != workflows: orchestrating data with Dagster
summary
Claims that Dagster replaces workflow-focused Airflow by treating pipelines as graphs of persistent data assets rather than task DAGs and shows concrete features: visual asset pages with lineage, a materialize button that runs only required assets, integration with DBT model-level assets, and automatic de-duplication so hourly and daily consumers avoid redundant runs. It demonstrates scheduling by freshness policies and a concrete asset versioning system that flags stale downstream assets after code or source changes and lets only those be re-materialized.
AI-written summary · about
chapters (14)
- 0:00Data pipelines vs Dagster
- 2:11Workflow engine limits like Airflow
- 3:35Defining Dagster data assets
- 5:41DBT models as Dagster assets
- 7:26Fallback to task workflows
- 8:08Visualizing assets in UI
- 9:10Monitoring assets in production
- 10:35Tracing asset lineage
- 11:38Scheduling from asset view
- 13:23Imperative scheduling challenges
- 16:11Freshness policies for updates
- 17:37Versioning code and data changes
- 19:02Granular asset versioning
- 21:08Summary of Dagster’s asset approach
related talks
36mElegant data pipelining with Apache Airflow
35mOrchestrating success
19mDemystifying Modern Data Pipeline Architecture
30mToo Big for DAG Factories?
26mDataOps: Accelerating Digital Transformation
38mUse Apache-Airflow to Build Data Workflows
32mOrchestrating Your Data Pipelines with Apache Airflow
38mFrictionless Data Pipelines