Migrating to Multi Cluster Managed Kafka - 0 Downtime
summary
Claim: Wix moves 1.5B messages/day across >10K topics and 100K leader partitions for 2000 microservices to Confluent Cloud with 0 downtime. Migration uses a scripted, fully automated pipeline: a replicator service consumes partitions and produces to target clusters while saving offsets mapping so consumers resume exactly where they left off. Gradual topic-by-topic migration, test-topic relay of real traffic, and an orchestrator that can stop and rollback make the process safe at scale.
AI-written summary · about
chapters (19)
- 0:00Multi-cluster Kafka migration intro
- 1:28Why Wix left self-hosted Kafka
- 3:14Managed Kafka benefits
- 4:39Greyhound Kafka client
- 6:03Cluster split by DC and SLA
- 6:45Choosing Kafka cluster with Greyhound
- 7:27Migration challenges and plans
- 8:52Live traffic gradual migration
- 10:17Zero-loss replication service
- 11:43Migration orchestrator
- 13:29Producer migration and rollback
- 14:11Test topics migration
- 14:53Migration monitoring dashboards
- 15:35Replication bottlenecks war story
- 16:58Christmas Eve Kafka bug war story
- 18:43Fixing Kafka config bug
- 19:24Scaling replication with consumer groups
- 20:07Post-migration Greyhound features
- 20:49Migration outcomes summary
related talks
35mKafka Deployment at Scale
41mA Future Journey: How to Migrate 100 Clusters Between Clouds Without Downtime?
19mZero-Downtime by Design: High-Availability Migration Patterns
24mZero Downtime Mastering Cassandra Live Migrations at Scale
13mScaling Kafka on Kubernetes
31mFrom Atlas to EKS How to Stay Sane and Save 45 During a Large‑Scale MongoDB Migration
29mKafka on Kubernetes: Does it really have to be the hard way?
04mPREVIEW: Running Kafka for Maximum Pain