Dancing With the Pods: Live Migration of a Database Fleet While Serving Millions of Queries
Talkat KubeCon + CloudNativeCon Europe 2025 · on Thursday, 2025-04-03· at 11:45· Level 1 | Hall Entrance S10 | Room A·
summary
ClickHouse migrates a legacy fleet of thousands of customer clusters over ~6 months with zero downtime by replacing single StatefulSets with a multi-statefulset “make-before-break” pattern that temporarily runs old and new replicas in parallel. A custom migration controller plus Temporal workflows coordinate batching, retries and rollbacks; fixes include zone pinning to avoid AZ skew, a system sync from modifier to avoid endless syncs, and an S3-backed disk type for zero-copy attaches — migration completes successfully at scale.
AI-written summary · about
chapters (13)
- 0:00Migration challenge
- 1:05ClickHouse architecture limits
- 3:12Vertical scaling breaks first
- 5:40Make-before-break scaling
- 7:27Zero-downtime requirements
- 9:56Temporal for migration workflows
- 11:43Migration orchestration
- 12:04Migration controller details
- 15:11Customer migration challenges
- 17:18ClickHouse migration issues
- 20:10DNS and system table issues
- 21:55S3 disks for system tables
- 22:37Live migrations Q&A
related talks
35mWhen StatefulSets are not enough
27mStateful Workloads at Scale – Extending Kubernetes for Running OLAP Databases as a Service.
32mMulti-Cluster Stateful Set Migration: A Solution To Upgrade Pain
35mZero-downtime deployment on Kubernetes
24mOptimizing K8s and unlock blockers with Pod Live Migration
38mMigrating From Single-Node Kubernetes Control Plane To HA In Product...
24mMigrations under Production Load: How to Switch...
37mZero downtime Deployment on Kubernetes