Lessons Learned From Etcd the Data Inconsistency Issues
summary
Covers the recent critical data-corruption bug in etcd (the distributed key-value store used by Kubernetes and other CNCF projects), explaining the root causes of the data inconsistency, how the bug was discovered, and the remediation steps taken to repair and protect clusters. Describes fixes applied to etcd releases and operational actions administrators had to perform, and extracts lessons learned for the cloud-native community.
AI-written summary · about
chapters (11)
- 0:00Start
- 1:33Postmortem
- 3:01Distributed consensus
- 4:51Raft log
- 9:42Causes
- 14:48etcd transaction vs BoltDB transaction
- 17:18Enhanced design which caused the issue
- 19:04Final compromised solution
- 19:57Three solutions comparison
- 22:30What went wrong
- 28:17Personal learnings
related talks
26mDon't Let Your Kubernetes Cluster Go Wild: Ensuring Etcd Reliability
19mDr etcd; or; How I Learned to Stop Worrying and Love the Datastore - Nick Young, VMware
23mThe Secret Recipe of etcd
20mLessons Learned from Operating ETCD
47mNot Omnipotent etcd, or Why It Doesn't Handle Heavy Loads
06mKeynote: Distributed Systems Simplified Using Kubernetes
42metcd: distributed locking and service discovery
28mHow CoreOS sees the container industry