Queues in PostgreSQL
summary
Argues PostgreSQL is a practical choice for moderate single-node job queues when work must be transactional and ops already manage the database. Recommends using FOR UPDATE SKIP LOCKED to avoid lock serialization and LISTEN/NOTIFY to wake workers, warns of thundering herd effects, B-tree churn, vacuum tuning, and 32-bit key exhaustion. Notes SERIALIZABLE isolation kills concurrency for queues, shows throughput improves with SKIP LOCKED, and proposes conveyor-belt storage or circular scans as future fixes.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:19What is a queue?
- 5:36Why put a queue in an RDBMS
- 7:32Example queue use cases
- 9:26How would you build a queue in Postgres
- 15:34Complications with queue processing in Postgres
- 19:09What could we do better to improve Postgres queue processing?
- 24:20Q&A with Thomas & Aaron
related talks
23mRethinking Message Queues: PostgreSQL's Role
29mPostgreSQL queues done right with PgQ
48mUsing Asynchronous Message Queues with PostgreSQL
44mBuilding a Distributed Queue with Delayed Jobs on PostgreSQL
23mPostgres as a Message Broker
48mUnderstanding and Reducing Application Contention
24mPostgreSQL Design Patterns
44mPushy Postgres and Python