Building Reliable, Scalable Services with Clojure and Core.async
summary
A Kafka-to-Elasticsearch bulk writer uses the Elasticsearch Java bulk API with asynchronous callbacks, which causes uncontrolled consumption, offset loss, and possible data loss; replacing ad-hoc callbacks with core.async channels and a go-fsm DSL connects a blocking data channel (backpressures Kafka) to a non-blocking control channel (bulk-start/stop, es green/red), enabling retries, exponential backoff, and safe shutdown. Implementation runs in production two years (≈300 rps per instance) and is testable by replaying channel events; code will be posted to GitHub.
AI-written summary · about
chapters (9)
- 0:00Introduction and context
- 1:54Problem identification
- 3:30Analyzing service failures
- 7:42Proposed technical solutions
- 10:43Implementing the loop
- 15:00Handling degraded states
- 17:44Demonstrations
- 19:25Borrowing FSM patterns
- 23:11Lessons and conclusions
