Implementing Celery, Lessons Learned
summary
Company moves from a Postgres-backed queue (running into millions a day, table bloat, and rare duplicate jobs) to Celery + RabbitMQ, adds separate high/medium/low queues and dynamic workers, and implements a single failure queue plus Sentry alerts. Producers delay enqueue until SQL Alchemy’s after commit event to avoid orphaned tasks; workers wrap tasks in DB sessions that commit on success and rollback on failure. PostgreSQL triggers use the pg_amqp extension (had to force JSON content type) to publish tasks.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:50Old queueing system
- 4:29Celery overview
- 7:16Multi-environment setup
- 9:03Task priority management
- 11:32Handling task failures
- 15:49Sentry integration
- 18:59SQLAlchemy integration
- 24:13Worker-side database management
- 27:15Postgres triggered tasks
- 29:26Task naming and triggers
- 34:00Q&A and conclusion
related talks
17mBuild a prod-ready distributed task queue system with celery
02h 45mThe world of High Performance Distributed & Async Task Queue(s) with Celery
23mPostgres as a Message Broker
23mRethinking Message Queues: PostgreSQL's Role
24mBuilding Workflows With Celery
30mA love letter to messaging: Celebrating RabbitMQ's journey
27mRuby Messaging Patterns
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)