Postgres as a Message Broker
summary
Postgres replaces RabbitMQ/Redis for async work by storing tasks in SQL, using Redis as a buffer and Lambdas to bulk insert 1k–2k rows, consumers fetching batches of ~10 with select for update skip locked, and a sidecar that marks abandoned tasks; runs ~200–300 workers, cuts queue infra cost ~50–60%, and drops RMQ RAM overloads. Solves high API latency with buffering, fixes queue starvation by isolating heavy queues, and reduces lock manager contention by removing unused indexes and shortening transactions.
AI-written summary · about
chapters (15)
- 0:00Introduction to Postgres as message broker
- 1:07Why switch from RabbitMQ
- 3:16PGQ introduction and features
- 5:03Producer and consumer workflow
- 7:52Producer reliability and retry logic
- 9:19Consumer flexibility and code snippet
- 10:00Cron runner and task scheduling
- 11:08Sidecar process for worker monitoring
- 11:50Final architecture overview
- 12:11High response time issue and solution
- 14:59Task starvation issue and fix
- 16:03Lock manager weight events problem
- 19:13Solutions for lock manager issues
- 20:39Performance optimization results
- 21:00Conclusion and benefits summary
related talks
23mRethinking Message Queues: PostgreSQL's Role
43mBlazingly Fast Message Queue on Postgres with Rust
48mUsing Asynchronous Message Queues with PostgreSQL
38mImplementing Celery, Lessons Learned
29mPostgreSQL queues done right with PgQ
50mScaling Postgres With Some Help from Redis
30mQueues in PostgreSQL
31mBatch Job / Queue Processing Without a Message Broker