How to scale a distributed (file) system
summary
Claim: avoid n×n config gossip by delegating coordination to a small leader set m (recommendation: m = 3 or 5, i.e. m << n), run a consensus-backed config store implemented with consul/etcd instead of full-mesh syncing. Practical result: prototype integration with Glusterfs shows much lower control-plane chatter, deterministic leader election and heartbeats, and far fewer split‑brain incidents. Tradeoffs noted: write latency and temporary unavailability during leader election, but overall scalability and operational simplicity improve.
AI-written summary · about
chapters (11)
- 0:00Start
- 13:15RAFT - A consensus algorithm
- 15:26RAFT: Terms
- 18:14RAFT: Server states
- 18:53RAFT: Replicated state machine
- 19:53RAFT: Different RPCS
- 20:55RAFT: Leader Election
- 23:12Consistent distributed store
- 26:57Why etcd
- 27:29Conclusion
- 30:14References
related talks
57mScaling the CAP Theorem for Modern Distributed Systems
52mThe Computer Science behind a modern distributed data store
28mTobias Petry :: Scaling Databases
28mWhy Are Distributed Systems So Hard?
20mLessons Learned from Operating ETCD
26mBetter Scalability & More Isolation? The Cortex “Shuffle Sharding” Story
42mScaling Resilient Systems: A Journey into Slack's Database Service
47mDatabase Resiliency