A DIY Guide to Building Data Pipelines with Python

on YouTube ↗23m

" summary

Shows a concrete DIY pipeline in Python that replaces Airflow for a scraping job and cuts end-to-end runtime from 24 hours to about 30 minutes by splitting work into micro-pipelines. Uses RabbitMQ + Celery for queued tasks, simple workers that download, parse and word-count articles, and stores logs/offsets in Mongo with bronze/silver/gold staging. Demo runs tasks in parallel on the first 10 articles, with a reproducible setup in Docker Compose (worker, queue, DB).

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 1:52Data engineering basics
3. 5:10Pipeline orchestration philosophy
4. 10:09Apache Airflow challenges
5. 13:45Building a simple pipeline
6. 16:22Scraping and processing
7. 21:11Next steps and resources
8. 21:31Q&A and closing

" related talks

• 34mData pipelines powered by Open source and funSayantika Banik · stackconf 2022
• 45mDefining data pipelinesJuan Riaza · Commit Conf 2018
• 43mData Pipelines à la ModeTom Hall · Code Mesh LDN 2019
• 34mData Ingestion in Python using Apache AirflowSowmya Ganapathi Krishnan, Yew Wee · PyCon APAC 2018
• 22mOrchestrate your data pipelines using Apache AirflowMahmoud Fettal · BlaBlaConf 2022
• 29mDagster: a Framework for Data Processing ApplicationsNick Shrock · Scale By The Bay 2019
• 35mBecome a Conductor with Apache AirflowMarwen Ben Dhahbia · Devoxx France 2021
• 25mBuilding a Python Data Pipeline with Apache FlinkCaito Scherr · PyCascades 2021