Batch Job / Queue Processing Without a Message Broker
summary
Tolgee replaces external brokers by queuing jobs in PostgreSQL and using row locks to avoid double-processing, running workers inside the application (Kotlin/Spring Boot) and coordinating nodes with Redis pub/sub and caching. Jobs run inside a transaction with savepoints so failures update job state safely; a blocking queue abstracts local vs Redis-backed queues. Measured throughput is about 3,000 jobs/s without Redis and 2,000 jobs/s with Redis, and the system handles chunking, retries and websocket progress updates.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:27Batch jobs in TLG
- 4:48No Kafka or RabbitMQ
- 8:25TLG deployment options
- 10:17Redis in solution
- 12:50Demo setup and Spring Boot
- 13:55Job execution service
- 16:53Blocking queue and Redis Q
- 17:35Consistency with FOR UPDATE SKIP LOCKED
- 19:24Testing job locking
- 20:08Performance comparison
- 21:54Job status with Redis PubSub
- 23:19Resources and closing
- 24:33Q&A
- 26:22Q&A: Implementation
related talks
23mRethinking Message Queues: PostgreSQL's Role
23mPostgres as a Message Broker
29mPostgreSQL queues done right with PgQ
30mQueues in PostgreSQL
44mBuilding a Distributed Queue with Delayed Jobs on PostgreSQL
26mSkip the Queue! Event-Driven Apps for Everyone with Postgres!
43mBlazingly Fast Message Queue on Postgres with Rust
48mUsing Asynchronous Message Queues with PostgreSQL