Plumbing in Python: Pipelines for Data Science Applications
summary
Blue Yonder builds and runs a Python data‑flow library called llamar that declares pipelines (splitter/reducer/fork) separately from execution, uses dill to pickle functions and data, and runs on local, subprocess or internal cluster back ends; projects hit 500 billion automated decisions per month and report faster iterations. The system adds pipeline observers (graph and performance), fuses linear segments to avoid serialization, prototypes explicit IO annotations, and is currently not yet open source.
AI-written summary · about
chapters (7)
- 0:00Pipelines in data science
- 2:32Pipeline structure and benefits
- 9:04Advanced pipeline constructs
- 15:30Yama library implementation
- 22:03Development and architecture impact
- 27:49Future plans and optimizations
- 35:19Q&A
related talks
38mBuilding Data Pipelines in Python
34mpydiverse pipedag: A library for data pipeline orchestration
28mYour Pipeline as a Function.
38mFrictionless Data Pipelines
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
24mLuigi & Data Pipelines
23mWhat would the django of data pipelines look like? with Lisa Dusseault
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python