Replacing Makefiles with Python and Luigi
summary
Luigi is used at CARTO to manage data ingestion pipelines; the talk introduces Luigi and discusses practical pros and cons not covered in the manual. It shares concrete tips and anti-patterns learned in production to help decide whether Luigi fits your workflow and how to avoid pitfalls that make it hard to use.
AI-written summary · about
chapters (12)
- 0:00Luigi introduction
- 1:04What is Luigi?
- 2:08Makefiles comparison
- 3:51Luigi practical example
- 5:14When to use Luigi
- 6:38Luigi advantages
- 8:02Luigi real experience
- 9:27Common problems and fixes
- 11:52Dynamic parameter tasks
- 13:56Overriding complete method
- 14:40Update alternatives
- 15:42Q&A
related talks
24mLuigi & Data Pipelines
35mMachine Learning Pipeline using Luigi and Scikit Learn
38mBuilding Data Pipelines in Python
42mLessons from 6 months of using Luigi in production
28mDynamic Data Pipelining with Luigi
32mLarge-scale machine learning pipelines using Luigi,...n
38mBuilding Data Pipelines in Python
44mCreating Python Data Pipelines in the Cloud