Dagster: a Framework for Data Processing Applications
summary
An open-source Python framework called Dagster models pipelines as graphs of typed functions and ships a local IDE, auto-completing type-checked config, live logs and event metadata. Computations are solids that can be Spark, SQL or plain Python, run on Airflow/Dask/Kubernetes, and support intermediate shuffling so failed runs retry from the last successful step. The claim is improved developer lifecycle: unit-testable pipelines, incremental execution, cron-based scheduling, GraphQL API and pluggable environments for consistent local-to-prod behavior.
AI-written summary · about
chapters (15)
- 0:00Introduction and motivation
- 3:34Problems in data pipelines
- 7:46Dagster's approach and design
- 10:40Hello world demo
- 12:24Pipeline visualization and metadata
- 13:31Configuration and execution
- 14:53Local development environment
- 16:21Debugging and error handling
- 17:25Incremental computation
- 18:31Testing pipelines
- 20:37Scheduling and production
- 21:39Testing strategies
- 24:11Metadata and composability
- 25:37Cross-system integration
- 27:01Deployment and ecosystem
related talks
47mDagster: An open source Python library for building data applications
45mDefining data pipelines
46mUse frameworks to test Apache Airflow data pipelines
34mData Ingestion in Python using Apache Airflow
40mPyData Stack: Pure Python open source data platforms
32mHuman-Friendly, Production-Ready Data Science Stack With Metaflow & Kubernetes
27mModern ETL-ing with Python and Airflow (and Spark)
21mSoftware Best Practices for Data Pipelines: Form Energy's Multi-Day Storage R&D Pipeline 🌳