Diving into ElasticSearch Discovery #bbuzz
summary
ElasticSearch discovery actually spans node discovery, leader election, state publishing and failure detection; the uploader-built ESCA (an Akka-cluster plugin) uses Akka seed nodes and the cluster singleton for leader election, asynchronous remoting to publish binary cluster state, and a seed-node quorum vote to mark unreachable nodes. Found surprising behaviors: ES 1.2 could see multiple masters, most updates use a single-threaded update-task queue, acknowledgements are weaker than expected, and ES 2.0’s ability to publish diffs should reduce large-state bottlenecks.
AI-written summary · about
chapters (9)
- 0:00Introduction to Etsy search
- 1:01Cluster coordination models
- 2:58Akka cluster discovery
- 3:47Elasticsearch cluster state
- 5:52Core discovery responsibilities
- 8:05Leader election mechanisms
- 9:39State publishing and transport
- 11:33Failure detection and handling
- 14:17Consistency and conclusions
related talks
44mElasticsearch at Scale of Billions of Documents
23mReaching Zen in ElasticSearch's Coordination
45mIs there anybody out there?
31mAKKA CLUSTER MANAGEMENT AND SPLIT BRAIN RESOLUTION
41mHands on Introduction to Distributed Systems Concepts with Akka Clustering
19mYou Know, for Search. Querying 24 Billion Records in 900ms
53mDistributed Eventsourcing with Akka
24mAkka - cluster - etcd