Creating Python Data Pipelines in the Cloud
summary
Compares Luigi, Airflow and AWS Data Pipeline by building a concrete pipeline that fetches Strava‑style logs, writes raw files to S3, runs ETL into RDS/Redshift, aggregates and produces a report; shows cron + bash fragility versus Luigi’s Python task/target model, Airflow’s DAG scheduler, metadata DB and Celery executors, and AWS’s UI/JSON configuration with tight vendor coupling. Claims Luigi is mature and lightweight, Airflow is feature‑rich but complex, and AWS pipeline trades programmability for console convenience.
AI-written summary · about
chapters (21)
- 0:00Cloud data pipelines intro
- 1:49Introduction
- 2:11What is a data pipeline
- 4:40Cycling data pipeline example
- 8:34Naive pipeline management
- 12:05Workflow management tools
- 13:10Luigi overview
- 16:41Luigi tasks and targets
- 18:06Implementing Luigi tasks
- 21:04Advantages and limitations of Luigi
- 23:14Airflow introduction
- 25:23Airflow components
- 27:11Airflow operators
- 29:17Defining Airflow DAGs
- 32:08Dynamic Airflow tasks
- 35:29Advantages and disadvantages of Airflow
- 36:33AWS Data Pipeline
- 38:01AWS Data Pipeline UI
- 40:29Luigi vs Airflow vs AWS
- 41:54Choosing a workflow tool
- 42:58Q&A
related talks
38mBuilding Data Pipelines in Python
38mBuilding Data Pipelines in Python
45mDefining data pipelines
46mGetting Started With Data Pipelines Using Apache Airflow
24mLuigi & Data Pipelines
32mLarge-scale machine learning pipelines using Luigi,...n
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
36mElegant data pipelining with Apache Airflow