Dynamic Data Pipelining with Luigi
summary
Shows Luigi building a file-driven ETL: watch a remote filesystem, dynamically require tasks at runtime to parse CSV or JSON, transform into a standardized table, then load into Postgres using Luigi’s CopyToTable Postgres target. Checkpoints are just task outputs so pipelines resume where they left off (checkpointing via outputs). Example pipeline is ~100 lines of code, uses local/S3/GCS/Postgres targets, handles idempotency but does not include a scheduler — external cron/Celery needed — and Luigi’s UI is minimal.
AI-written summary · about
chapters (8)
- 0:00Dynamic Data Pipelining with Luigi
- 2:39Pipelining 101
- 4:37How about ETL?
- 6:48Luigi Crash Course
- 11:44A Real World Example
- 17:35Alternatives
- 19:52Sources
- 20:39Questions?
related talks
24mLuigi - Big data plumbing library
43mData Pipelines à la Mode
51mDeclarative ETL pipelines with Delta Live Tables
35mMachine Learning Pipeline using Luigi and Scikit Learn
39mData Pipeline Evolution
16mDemocratizing Data Pipelines: Introducing the AI-Powered ETL Assistant
30mDesigning Resilient Data Pipelines
42mEnd-to-end pipeline agility