Dagster: An open source Python library for building data applications
summary
Dagster is an open source Python library for building ETL, ELT and ML pipelines as graphs of typed computations. It models nodes as solids with inputs, outputs and a unified type and configuration system, persists intermediates so runs can re-execute individual steps, and exposes testing APIs for local unit and integration runs. A lightweight cron-based scheduler ships with it, it can compile pipelines into Airflow DAGs, and execution is pluggable across runtimes like Dask, Kubernetes or Spark.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 0:45What is Dagster
- 12:12Dagster
- 15:12Data Tools
- 18:11Testing
- 21:21Scheduling
- 22:55Testability
- 26:40Frame the goal
- 27:47Airflow doesnt address development and test
- 28:28Airflow tasks
- 29:30XCOM
- 33:04Metadata
- 34:54Airflow Dag
- 36:14Multidisciplinary pipeline
- 37:00Data application representation
- 38:48Integrations
- 39:58Summary
- 40:19Thank you
- 40:49QA Stable or mature
- 41:45Metadata and data
- 42:50Scheduler
- 43:30One Point
- 44:05Conversion
- 44:27Pipeline states
- 45:05Serializing data between solids
related talks
29mDagster: a Framework for Data Processing Applications
40mPyData Stack: Pure Python open source data platforms
35mDagobah, a Data centric Meta scheduler
34mData Ingestion in Python using Apache Airflow
34mData pipelines powered by Open source and fun
05mLightning Talk: Ditching Data Pipelines: why treating Data as Assets is the best...
46mUse frameworks to test Apache Airflow data pipelines
22mOrchestrate your data pipelines using Apache Airflow