Creating a Distributed Round Robin Scheduler with Etcd
summary
Built a distributed round robin scheduler on top of etcd using leases and watchers and embedded etcd in a Go service, avoiding a single leader and instead having multiple schedulers compete to advance a lexicographic ring. Implementation uses etcd transactions, lease keepalives and gRPC watch fanout, scales to thousands of workers per scheduler, and leverages the etcd 3.4 lease heap fix after hitting slow lease expiry. Testing and failure injection proved hard but the system works in production.
AI-written summary · about
chapters (7)
- 0:00Start
- 2:24Scheduling Mode
- 11:44Diagram of the Replicated State Machine Architecture
- 17:09Key Takeaways
- 24:09Leases and Watchers
- 25:12Create a Distributed Trigger
- 33:54Does It Scale
related talks
26mDon't Let Your Kubernetes Cluster Go Wild: Ensuring Etcd Reliability
42metcd: distributed locking and service discovery
20mLessons Learned from Operating ETCD
40mDeep Dive: etcd
06mKeynote: Distributed Systems Simplified Using Kubernetes
28mHow CoreOS sees the container industry
34mPatterns for building Highly Available Cluster Schedulers
40mDeep Dive: etcd