Cluster Consensus: When Aeron Met Raft
summary
Cluster consensus combines Raft semantics with Aeron messaging to get a byte-stream, pipeline-driven implementation that orders inputs, appends immutable logs, and commits once a majority persists entries. It uses off-heap byte buffers, spy-based subscribers to avoid copies, multicast-emulation via multi-unicast, monotonic counters for positions, and a timer wheel offering nanosecond timers. Performance claims include sustained 2 GB/s over 10GbE for log replay; a working implementation exists with snapshotting and cluster catch-up features under development.
AI-written summary · about
chapters (46)
- 0:00Intro
- 3:03What does consensus mean
- 3:16Secondary definition of consensus
- 3:33What is consensus
- 4:52Raft
- 5:57What is Raft
- 8:23How the election works
- 10:48Safety guarantees
- 11:33Functional theory
- 12:14State as immutable
- 12:31Why Aeron
- 12:54Guaranteed Delivery
- 13:43NeedtoKnow Approach
- 14:06Guaranteed Processing
- 15:09Single Service
- 17:51Open Source
- 18:07Do servers crash
- 19:02Weird Java code
- 20:13Performance
- 20:25RPC
- 22:08concurrency and parallelism
- 23:45hard work
- 24:48Speculation engines
- 25:20Pipelining
- 25:36Consensus
- 25:54Pipeline
- 26:26Example
- 28:05bytebuffers
- 29:21bug
- 29:37message index
- 30:14byte index
- 31:12Multicast
- 33:03Monotonic functions
- 33:38Efficiency
- 35:02Timers
- 36:01Scheduling Timers
- 36:13Scheduling Timer API
- 36:32Timer Wheel
- 36:57Complexities
- 37:36Real Code
- 37:54Session Status
- 38:19Continued Action
- 39:24Distributed File System
- 40:20Processing
- 42:51Implementation
- 43:07Roadmap
related talks
14mDistributed Consensus in 15 Minutes!
47mDistributed Systems: When Democracy and Computer Science Come Together
47mUnderstanding Raft Consensus
41mThe Raft Consensus Algorithm
55mImplementing Raft in RabbitMQ
43mRa: a Raft implementation
50mDo Distributed Systems they said. It'll be fun
01h 32mHow to Reach Consensus with Paper, Pencil and the Raft Algorithm