Designing Concurrent Distributed Sequence Numbers ...
summary
Explains why sequence numbers for ordering let ElasticSearch replace heavyweight consensus with a practical master‑backup design, evaluates Raft/ZAB and finds their single‑threaded, quorum‑ack constraints unacceptable for concurrent Lucene indexing, and proposes replicating to all copies then broadcasting a safe commit point so replicas expose only durable changes. Describes using existing synchronous replication, measuring rollback frequency tradeoffs, and building a lightweight commit marker that lets replica sync transfer a few operations instead of gigabytes. master‑backup replication
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:40Document level versioning
- 1:33Multiple doc updates - with seq#
- 2:05Sequence # == ordering of changes • meaning they can be sorted, shipped, replayed
- 2:42Primary Replica Sync
- 3:50Primary Replica File Based Sync
- 4:12Primary Replica Seq# Based Sync
- 5:59Concurrent Indexing
- 8:31Raft - appendEntries
- 9:25Raft - commit on quorum
- 10:32Raft - primary failure
- 10:55Raft - ack on quorum
- 14:32Master-Backup - indexing
- 17:443 histories
- 21:27Final words
related talks
52mThe Computer Science behind a modern distributed data store
38mTowards consensus on distributed consensus #bbuzz
41mRule #2: Double Tap. An Elasticsearch Journey of Resiliency
47mDatabase Resiliency
51mDo Not Reimplement Distributed Consensus Algorithms
44mDistributed systems and scalability rules
48mHow to build distributed concurrency primitives
40mConsensus Algorithms in Distributed Systems