Running distributed stateful system on Kubernetes - Beyond Statefulsets
summary
StatefulSets work for small deployments but break at scale (tens–hundreds of nodes) due to ordinal reuse causing PV reattachment risks, lack of eviction/PDB honor, enforced homogeneity, and immutable PV specs. Built a custom controller — cockroach-operator — with two CRDs (CockroachDB cluster and CockroachDB node) that implements surgical rolling restarts, random UIDs to avoid PV reuse, explicit PVC lifecycle linking, per-node certificates via lifecycle hooks, and operator-level PDB coordination; result: stable, scalable CockroachDB operations across clouds.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:42StatefulSets purpose
- 3:10StatefulSets for stateful apps
- 5:15StatefulSets limitations
- 6:20CockroachDB architecture
- 9:10CockroachDB on Kubernetes
- 10:34Running CockroachDB with StatefulSets
- 12:20Why moving beyond StatefulSets was necessary
- 13:24Custom operator design
- 14:26Operator challenges solved
- 16:53Pod Disruption Budgets handling
- 17:55Node homogeneity and security
- 18:58Immutable storage fields
- 19:20Conclusion
related talks
25mStatefulsets on Kubernetes: The Guardians of Your Data Galaxy
33mWhat We’ve Learned Building a Multi-Region DBaaS on Kubernetes
48mBuilding mission-critical applications with CockroachDB
26mChallenges of Managing Stateful Applications in Kubernetes
35mWhen StatefulSets are not enough
20mDeployments vs StatefulSets vs Daemonsets
45mDatabases on Kubernetes: Why you should care
32mDon’t Hassle Me, I’m Stateful