Playing tasks with Django-Celery
summary
Shows a working Django-Celery setup at Jamendo: RabbitMQ broker, periodic tasks stored in the DB and scheduled via the Django admin, and a celerycam that snapshots task state every 5–10s. Demonstrates splitting a 200000-track job into 200000 tracks split into 1000-item slices and dispatching subtasks so failures are isolated, explains retry behavior (countdown, exception passed to retry, default max_retries ≈30), and adds a custom Task base overriding call/after_return for temp folders and locks stored in MongoDB with manual release. Also adds centralized logging by attaching handlers to Celery’s log signals and a web button to rerun tasks from the admin.
AI-written summary · about
chapters (10)
- 0:00Introduction and Context
- 3:18Technical Difficulties
- 13:54Celery Task Basics
- 17:44Task Sets and Parallelism
- 22:08Custom Task Classes
- 24:17Django Celery Integration
- 28:50Manual Task Control
- 30:27Unique Task Management
- 33:52Centralized Logging
- 37:02Q&A and Final Remarks
related talks
40mHow to Schedule Tasks with Celery and Django
32mFrom crontab to celery with no regrets
49mIntroduction to Celery
25mMixing reliability with Celery for delicious async tasks
24mBuilding Workflows With Celery
28mBoosting Python Web Apps performance with Celery and Concurrency
19mOptimizing Data Retrieval with Python Celery
36mAn Introduction to Distributed Task Queue