Lessons Learned from Operating ETCD
summary
OVHcloud runs etcd under a Kubeception pattern and reports concrete ops rules: enable raft and disk metrics, tune timeouts and watch leader transfers and proposal/commit/apply latency, use SSDs with proper fsync and avoid slow disk stacks, keep individual etcd data under about 17 GB, automate hourly compaction and snapshotting so member rejoin is fast, prefer smaller clusters, and script safe member removal and replacement to avoid long catchup windows and cascading elections.
AI-written summary · about
chapters (14)
- 0:00Operating etcd in production
- 1:52What is etcd and consistency
- 3:21Kubernetes with etcd and CRDs
- 5:08Observability in etcd ops
- 7:20Key metrics for etcd health
- 8:49Leader election performance
- 10:17Raft consensus metrics
- 11:26Disk I/O and write perf
- 12:58Value store limits
- 14:03Database compaction
- 15:54Memory mapping issues
- 17:19Cluster disconnections handling
- 18:23Production failure lessons
- 19:28Quick etcd performance tips
related talks
23mThe Secret Recipe of etcd
19mDr etcd; or; How I Learned to Stop Worrying and Love the Datastore - Nick Young, VMware
06m⚡ Lightning Talk: Back To Basics: How To Measure Etcd Performance And Not To Die Trying
46mEtcd 101: The Parts They Don't Teach You
26mDon't Let Your Kubernetes Cluster Go Wild: Ensuring Etcd Reliability
06mKeynote: Distributed Systems Simplified Using Kubernetes
47mNot Omnipotent etcd, or Why It Doesn't Handle Heavy Loads
42metcd: distributed locking and service discovery