From crontab to celery with no regrets
summary
A company migrates hundreds of messy cron jobs into a Celery-based system and proves it handles backups, large imports/exports, session cleanup and long CPU tasks reliably. Architecture uses Celery + Redis broker, Django tasks exposed as shared_task, celery beat + Django periodic tasks driving schedules, and Flower for monitoring. Deployment runs via Docker Compose deployment with multiple worker replicas (they commonly run ~10 workers). Setup is fiddly but the GitLab repo contains runnable examples.
AI-written summary · about
chapters (10)
- 0:00Introduction and context
- 2:52The case for leaving cron
- 5:13Introduction to Celery
- 6:55Use cases for background tasks
- 10:04Architecture overview
- 12:06Broker selection
- 14:06Hello world example
- 17:16Integrating Celery with Django
- 22:12Scheduling tasks with Celery Beat
- 28:53Containerization and conclusion
related talks
45mPlaying tasks with Django-Celery
40mHow to Schedule Tasks with Celery and Django
49mIntroduction to Celery
45mPlaying tasks with Django-Celery
42mCelery on AWS ECS - art of background tasks & continuous deployment
28mBoosting Python Web Apps performance with Celery and Concurrency
30mBuilding data flows with Celery and SQLAlchemy
29mSwiss Army Django: Small Footprint ETL