When StatefulSets are not enough
summary
Kubernetes StatefulSets block safe vertical resizing and zero-downtime autoscaling for ClickHouse Cloud, so the team builds a make-before-break flow by replacing one monolithic StatefulSet with per-pod multi-STS objects, adds zone‑pinning and PVC snapshot guards, and runs coordinated migrations via Temporal workflows plus a wrapper migration controller. The migration moves thousands of clusters over a year with no customer downtime, required core DB fixes (replica sync FROM modifier, materialized view and external engine resilience) and proved the approach in production.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:03What is ClickHouse
- 3:07ClickHouse operator basics
- 6:20Vertical scaling challenges
- 9:52Break-first scaling limitations
- 13:00Evaluating existing solutions
- 15:49Multi-stateful set concept
- 18:14Migration challenges
- 22:49Migration controller design
- 25:37Topology spread constraints
- 27:45Zone pinning solution
- 28:27Database elasticity issues
- 30:35Unbreakable replicas problem
- 31:39System table synchronization
- 32:00Fleet migration journey
- 33:04Conclusion
related talks
27mStateful Workloads at Scale – Extending Kubernetes for Running OLAP Databases as a Service.
23mDancing With the Pods: Live Migration of a Database Fleet While Serving Millions of Queries
32mMulti-Cluster Stateful Set Migration: A Solution To Upgrade Pain
25mStatefulsets on Kubernetes: The Guardians of Your Data Galaxy
22mRunning distributed stateful system on Kubernetes - Beyond Statefulsets
32mDon’t Hassle Me, I’m Stateful
05mUpgrading Stateful Sets in Constrained Environment
41mData warehouse and Kubernetes: Lessons from ClickHouse Operator