Kafka based Global Data Mesh at Wix.
summary
Wix runs Kafka at massive scale: about 70 billion events/day, ~50k topics and ~500k partitions across 18 data centers and 2,300 microservices. Developers use an in-house abstraction called Greyhound (JVM lib + gRPC sidecar clients) that offers resilient producers, parallel and batch consumers, context propagation, and metrics. Multi-cluster deployment splits traffic by flows (editor vs visitors), replication uses a compact topic replication config and a batch replication service, and Confluent tiered storage backs Kafka with S3 to avoid local disk exhaustion.
AI-written summary · about
chapters (14)
- 0:00Introduction to Wix's Kafka architecture
- 4:24Kafka basics and Wix's scale
- 6:55Introducing Greyhound library
- 9:05Greyhound features for developers
- 11:57Greyhound as a service
- 14:30Creating a global data mesh
- 15:13Managing growing Kafka clusters
- 17:27Simplifying cluster interaction
- 19:55Cross-data center communication
- 22:28Replicating data between data centers
- 25:22Decoupling producers and consumers
- 27:52Future of global event-driven architecture
- 30:26Summary and Greyhound resources
- 32:33Q&A session
related talks
57mOur DataMesh Journey: data architectures for microservices
26mKafka in the Wild
40mA Data Mesh Of Streams
41mKafka as a Platform: The Ecosystem from the Ground Up
39mKafka as a Platform: The Ecosystem from the Ground Up
45mRethink Data Accessibility: Data Mesh meets Enterprise Integration
01h 07mKafka Streams and KSQLDB
19mSupercharging Kafka Applications with Real-time Contextual Insights