Building Data Pipelines in Python
summary
Argues that reproducible, production-ready pipelines need concrete practices: unit tests and end-to-end tests early, Python workflow managers (demonstrates Luigi with its task/target API and web graph), logging instead of prints, packaging small components, and automation. Gives real examples (NLP preprocessing and recruitment-profile matching), notes Luigi suits batch jobs up to tens of thousands of tasks/day, shows small amount of glue code required, and explains failure recovery and common anti‑patterns like “script soup.”
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:12R&D to production pipelines
- 4:00Good pipeline principles
- 6:49Avoiding script soup
- 7:58Unit testing importance
- 15:57Luigi workflow manager
- 19:27Logging in pipelines
- 22:17Bridging R&D and engineering
- 27:53Scaling Luigi for batch
- 30:15End-to-end testing
- 33:34Luigi pipeline examples
- 37:09Q&A
related talks
32mBuilding Your First Data Pipelines
44mCreating Python Data Pipelines in the Cloud
24mLuigi & Data Pipelines
25mTesting Data Pipelines
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
36mPlumbing in Python: Pipelines for Data Science Applications
27mThe necessity of pipelines in Natural Language Processing
32mLarge-scale machine learning pipelines using Luigi,...n