Reproducible Data Pipelines Using Controlled Effects
summary
Presents a Haskell architecture (kernmantle) that encodes side effects as binary abstract effects with two type parameters, composes tasks via arrows, and interprets effects by dependency injection into a named effect mantle so handlers run at config time. Demonstrates a Docker/PyTorch pipeline that downloads models and builds images during config-time, failing early when a model is missing. Claims reproducible, cacheable pipelines, static-graph only, supports local caching and test vs production handler sets, and shows the approach works on real classification examples.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:57About Twig
- 1:40Task Workflow vs Data Pipeline
- 3:12Tools Targeting Fine Grain Tasks
- 3:34Static vs Dynamic graphs
- 4:15Process Time vs Configuration Time
- 5:16Commental Approach
- 6:04Terminology
- 7:25Effects
- 9:38Problems
- 10:57Abstract Effects
- 14:28Classification Effects
- 18:01Triggering Effects
- 19:28Rope
- 22:13Rope Analogy
- 22:48Parameter Interpretation
- 23:38Mapping
- 25:30Example
- 27:30Running Example
related talks
43mData Pipelines à la Mode
30mSMART DATA PIPELINES FOR AI
35mScalable data pipelines with shapeless and cats
28mA Series of Unfortunate Effects
27mLimiting side-effects of applications at compile-time
35mSimplification of Data Pipelines by Containerization
30mDesigning Resilient Data Pipelines
20mTime Travel and Provenance for Machine Learning Pipelines