Luigi & Data Pipelines
summary
Luigi actually shows how to turn Jupyter work into reproducible pipelines by declaring Luigi tasks and targets that map sources to outputs so runs resume from missing files rather than restarting everything, with wrappers for Postgres BigQuery and Hadoop and a local Luigi daemon UI for visualizing graphs. Demo code lives at the uploaded GitHub repo and the workshop walks through building pipelines over the City of Chicago building permits dataset covering every permit from 2006 to present with a hands on mob programming session.
AI-written summary · about
chapters (9)
- 0:00Luigi and data pipelines
- 1:04Luigi origins
- 2:53Why use Luigi
- 5:47Luigi tasks and targets
- 9:41Basic Luigi task
- 17:31Luigi visualization and deployment
- 18:14Mob programming exercise
- 20:07Chicago building permits analysis
- 22:41Group assignment
related talks
32mBuilding Your First Data Pipelines
01h 27mProduction-ready Data Science with Python and Luigi
38mBuilding Data Pipelines in Python
38mBuilding Data Pipelines in Python
44mCreating Python Data Pipelines in the Cloud
32mLarge-scale machine learning pipelines using Luigi,...n
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
42mLessons from 6 months of using Luigi in production