The world of High Performance Distributed & Async Task Queue(s) with Celery
summary
A hands‑on Celery tutorial builds a working end‑to‑end banking notification system that sends SMS and WhatsApp via a task queue, uses RabbitMQ as the broker (Redis shown as an alternative), and wires Flower → Prometheus → Grafana for metrics. It demos .delay versus delay vs apply_async semantics, signatures/groups/chains, worker forking equal to CPU cores, autoscaling, bounded tasks and a custom state machine for production state tracking, and the examples run successfully.
AI-written summary · about
chapters (15)
- 0:00Celery and task queues
- 15:43Distributed async queues
- 21:34Celery setup prerequisites
- 39:52Basic Celery tasks
- 49:54Parallel processing with workers
- 57:12Auto-scaling workers
- 1:03:38Task execution delay
- 1:18:26Two-way communication
- 1:25:26Task chaining and signatures
- 1:42:19State management in tasks
- 1:55:55Managing queues
- 2:06:39Monitoring with Flower
- 2:21:36Distributed Celery nodes
- 2:32:27Redis as broker
- 2:43:02Summary and Q&A
related talks
17mBuild a prod-ready distributed task queue system with celery
24mBuilding Workflows With Celery
38mImplementing Celery, Lessons Learned
47mBackground Processing with
29mThe Hare Wins the Race: Getting the most out of RabbitMQ in distributed applications
41mDecouple all the things: Asynchronous messaging keeps it simple
01h 05mScaling Applications with RabbitMQ
46mReal-world Examples of Message Queue Usage: Insights from RabbitMQ