Building resilient scheduling in distributed systems with Spring
summary
Shows a concrete solution for running one scheduled job across scaled Spring instances: first uses an AOP run-if-leader annotation that queries an external orchestrator, then replaces it with a programmatic SchedulingConfigurer that registers Runnable tasks, a custom scheduled thread pool (tested at pool size 4), and a leader-aware decorator. Adds idempotency IDs to avoid split‑brain duplicates, evaluates Postgres row locking and SQS (14‑day visibility limit), and reports that the programmatic approach improves testability, observability and resilience.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 0:50Agenda
- 1:34How our systems are developed
- 3:53Simplified view of the problem
- 6:11Spring application example
- 7:32Vertical vs horizontal scaling
- 9:15Running scheduling on a single note
- 11:17Introducing another component
- 12:23Adding a leader
- 14:48AspectOriented Programming
- 16:38Advices
- 17:48Annotation
- 20:13Disadvantages
- 23:46programmatic approach
- 24:05interface
- 24:46code
- 26:14improvements
- 27:47benefits
- 30:49convenience of testing
- 31:10scheduling config test
- 34:49original problem
- 35:38other classes
- 36:36delegate
- 37:23split brain
- 40:05improve resiliency
- 44:26SQS queue
- 45:29Lessons learned
related talks
01h 00mScaling Batch Applications
32mJobRunr - distributed background processing using only Java 8 lambda's
01h 03mScaling Web Operations with Distributed Cluster Schedulers
44mDeep Dive talk | Resilience in Microservices with Congestion Control
34mPatterns for building Highly Available Cluster Schedulers
01h 01mLost in transaction Strategies to manage consistency in distributed systems
44mBuilding a Distributed Queue with Delayed Jobs on PostgreSQL
49mThe Benefits of Orchestration in Distributed Systems