Building a Distributed Queue with Delayed Jobs on PostgreSQL
summary
A developer shows how to build a distributed task queue using PostgreSQL and Java, with code samples for table/row structure in SQL and Java. The design uses Postgres locks and waiting for locks inside transactions to ensure retries and strict single-entity sequential execution; the talk also covers queue size, indexes, diagnostics for where the data came from, and load-test results.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:50Need for custom job queue
- 7:32Queue in PostgreSQL
- 12:35Selecting jobs with locking
- 15:53Batching by entity
- 20:49Index problems and fixes
- 27:25Partial indexes role
- 31:41Optimizing queue indexes
- 34:15Partitioning and cleanup
- 38:10Querying partitions
- 38:32Implementation limits
- 40:43Summary
- 42:31When to use this
related talks
30mQueues in PostgreSQL
23mRethinking Message Queues: PostgreSQL's Role
29mPostgreSQL queues done right with PgQ
48mUsing Asynchronous Message Queues with PostgreSQL
01h 05mDatabases: Bespoke Tailoring and Perfect Fit
23mPostgres as a Message Broker
26mSkip the Queue! Event-Driven Apps for Everyone with Postgres!
31mBatch Job / Queue Processing Without a Message Broker