Building Data Pipelines in Python
summary
Argues that the hard decisions are concrete: use Luigi to define task dependencies and targets, package pipeline steps as reusable Python modules, and run unit tests plus end-to-end checks so outputs are reproducible. Recommends structured logging over prints, asserts for quick notebook checks, and warns that workflow managers suit batch jobs up to tens of thousands of jobs per day. Shows a Slack/email integration and explains failure recovery and incremental reruns.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:09R&D versus engineering
- 3:19ETL and pipeline complexity
- 4:45Principles for good pipelines
- 6:53Script soup dangers
- 7:56Unit testing in Python
- 10:25Unit testing academic code
- 12:34Writing unit tests
- 15:45Testing libraries and tools
- 16:07Workflow managers like Luigi
- 19:41Logging importance
- 22:11Bridging R&D and production
- 23:58Q&A
related talks
44mCreating Python Data Pipelines in the Cloud
32mBuilding Your First Data Pipelines
25mTesting Data Pipelines
24mLuigi & Data Pipelines
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
45mDefining data pipelines
21mUsing DBT and Python to Build Production Pipelines in Snowflake
32mLarge-scale machine learning pipelines using Luigi,...n