Building Your First Data Pipelines
summary
Uses Luigi tasks and Targets to turn ETL steps into atomic file or database operations, with a local scheduler for quick runs and a web UI for dependency graphs. Builds a basic exploratory pipeline on DC open data with Jupyter notebooks, shows passing models and DataFrames via pickle files, and demonstrates S3, Postgres and Spark targets. Includes a hands‑on demo and a retrofit example that replaces brittle scripts with restartable, parameterized pipelines.
AI-written summary · about
chapters (14)
- 0:00Data pipelines overview
- 1:31Need for robust pipelines
- 4:00Luigi for pipelines
- 6:09Luigi workflow basics
- 8:36Luigi tasks and targets
- 12:31First Luigi task
- 17:37Database and cloud support
- 18:20Local scheduler setup
- 19:22Pipeline progress visualization
- 21:49Data frames and pickle files
- 23:15DC permits pipeline exercise
- 26:51Luigi with PostgreSQL
- 28:14Scaling to Spark
- 30:03Q&A
related talks
24mLuigi & Data Pipelines
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
38mBuilding Data Pipelines in Python
38mBuilding Data Pipelines in Python
32mLarge-scale machine learning pipelines using Luigi,...n
25mTesting Data Pipelines
46mGetting Started With Data Pipelines Using Apache Airflow
20mBuilding a Data Pipeline with Testing in Mind