Your Pipeline as a Function.
summary
A minimal Python pipeline models a jaffle shop using duckdb, S3 and Dagster concepts: pure-ish functions return DataFrames, memoization to S3 serializes cached results as CSVs to avoid redundant re-materialization, asyncio.gather runs independent upstream materializations in parallel, and hourly time partitioning treats each partition as immutable for easy backfills. GitHub examples show faster local tests, fewer network dependencies, cheaper reruns, and simple incremental recovery.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:40Functional programming basics
- 6:11Data pipeline challenges
- 9:01Current pipeline example
- 12:30Pure function approach
- 15:24Memoization benefits
- 18:38Caching and data changes
- 19:43Time partitioning solution
- 21:53Parallel execution
- 25:04Summary and resources
related talks
36mPlumbing in Python: Pipelines for Data Science Applications
34mpydiverse pipedag: A library for data pipeline orchestration
38mBuilding Data Pipelines in Python
31mBuilding 3D Trusted Data Pipelines With Dagster, Dbt, and Duckdb
21mDesigning Modular Data Pipeline
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
28mHow to not pull your hair out while providing data to the business
30mSpeed Up Your Data Processing