Task Failed Successfully
summary
Talk claims modern pipelines break when workflow systems treat failure as an outlier and documents concrete Airflow pitfalls: XComs are a cheat (admitted creator regret), an untraceable branch operator, unique-constraint on DAG runs, and persistent DB bloat. It proposes Prefect as a working alternative, open sourced, using Dask executor to run tasks, richer state objects so Prefect fails more successfully, and a focus on avoiding wrong-way risk rather than just promising examples.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:07Failing successfully
- 2:10Positive vs negative engineering
- 3:15Building task with negative engineering
- 5:42Workflow systems for dependencies
- 6:48Failure and system behavior
- 7:52Wrong-way risk in workflows
- 9:16Assumptions on perfection
- 10:21Three sins of workflows
- 12:04XComs unintended effects
- 14:53Branch operator dependencies
- 16:18Workflow state vs logic
- 18:06Airflow abuse patterns
- 19:29Workflow managers history
- 21:36Idempotency flaws
- 22:38Ideal workflow design
- 24:02Prefect features
- 25:47Prefect vs Airflow
- 26:32Q&A
related talks
25mHow NOT to Build a Pipeline
36mElegant data pipelining with Apache Airflow
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
22mData pipelines != workflows: orchestrating data with Dagster
29mMaking tech boring to keep data exciting
25mWhy Software Engineering Best Practices Fail in Data Engineering
27mModern ETL-ing with Python and Airflow (and Spark)
35mOrchestrating success