Building data flows with Celery and SQLAlchemy
summary
A working reporting pipeline built with SQLAlchemy core for bulk SQL and Celery Canvas chains and groups composes CSV extracts, staged transforms and loads into a central warehouse; processors are classes/mixins that create tables, derive columns and update rows, with per-column timestamps for incremental runs. Monitoring uses Celery events and Flower. One deployment cuts a four hour batch to an initial few minutes and then reduced runtime from four hours to seconds for incremental updates.
AI-written summary · about
chapters (30)
- 0:00Intro
- 0:30Coming up
- 2:24Why Data Warehousing?
- 2:45What is a Data Warehouse
- 3:35Extract, Transform, Load
- 3:46Python can help!
- 4:31Existing solutions
- 5:27Ways to move data around
- 6:13SQLAlchemy is...
- 7:15About SQLAlchemy
- 7:43Python support
- 8:21SQLAlchemy Core
- 8:47Create a table
- 9:08Insert data
- 9:28Query data
- 10:28Example Processor Types
- 10:39Abstract Processor
- 11:00Abstract Database Processor
- 11:19CSV Extract Mixin
- 11:51A Concrete Extract
- 12:34An Abstract Derive Transform
- 13:36A Concrete Transform
- 15:08Introducing Celery
- 16:04Canvas: Designing Workflows
- 17:25Sample Data Processing Flow
- 19:10Monitoring - celery events
- 19:59Monitoring - celery flower
- 20:14Turning it up to 11
- 22:12Summary
- 22:32Resources
related talks
23mBuilding a multi-purpose platform for bulk data using sqlalchemy
23mA DIY Guide to Building Data Pipelines with Python
32mFrom crontab to celery with no regrets
32mScalable Python and SQL Data Engineering without Migraines
28mBoosting Python Web Apps performance with Celery and Concurrency
19mOptimizing Data Retrieval with Python Celery
34mData pipelines powered by Open source and fun
49mIntroduction to Celery