The Secret Recipe of etcd
summary
etcd is the cluster metadata store and causes outages when its DB fills, revisions explode, or leaders keep reelecting. Practical actions include periodic compaction and defragmentation to free space, increasing disk size or DB defaults, and handling backups carefully (watch restore v3 vs v2 data for plugins). Slow disk or network causes leader churn so tune heartbeat interval and election timeout, prefer SSDs, monitor DB size/key count with Prometheus, and add members one at a time when recovering.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:52etcd overview and role
- 4:08Raft consensus algorithm
- 8:54etcd importance in Kubernetes
- 10:00etcd failure points and symptoms
- 12:57Recovering from leader loss
- 15:51etcd latency and Raft growth causes
- 18:23etcd tuning and configuration
- 20:10etcd monitoring with Prometheus
- 21:12Q&A
related talks
20mLessons Learned from Operating ETCD
19mDr etcd; or; How I Learned to Stop Worrying and Love the Datastore - Nick Young, VMware
06m⚡ Lightning Talk: Back To Basics: How To Measure Etcd Performance And Not To Die Trying
42metcd: distributed locking and service discovery
47mNot Omnipotent etcd, or Why It Doesn't Handle Heavy Loads
26mDon't Let Your Kubernetes Cluster Go Wild: Ensuring Etcd Reliability
46mEtcd 101: The Parts They Don't Teach You
31mDemystifying Kubernetes Failure Scenarios Caused by etcd