Build a prod-ready distributed task queue system with celery
summary
Shows how Grofers runs a production Celery fleet using Celery with Redis/RabbitMQ for broker and result backends, plus concrete tuning: worker concurrency, prefetch limits, rate limits, and routing rules. Describes failure handling with retry/backoff and idempotent tasks, autoscaling workers, and visibility via Prometheus and Grafana metrics and alerting. Demonstrates a deployed setup that recovers from broker and worker failures and sustains thousands of tasks per minute in real traffic.
AI-written summary · about
chapters (5)
- 0:00Introduction to Celery
- 3:12Distributed task system basics
- 8:11Optimizing Celery for production
- 12:17Handling system health and failures
- 16:02Conclusion and Q&A
related talks
02h 45mThe world of High Performance Distributed & Async Task Queue(s) with Celery
38mImplementing Celery, Lessons Learned
25mMixing reliability with Celery for delicious async tasks
42mCelery on AWS ECS - art of background tasks & continuous deployment
40mHow to Schedule Tasks with Celery and Django
24mBuilding Workflows With Celery
46mA Practical Introduction to Airflow
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow