Kafka Based Global Data Mesh at Wix

on YouTube ↗37m

" summary

Wix runs a Kafka heart with 2,200 microservices, 70 billion daily events, 50,000 topics and 500,000 partitions and claims a global data mesh that keeps producers and consumers decoupled. A JVM library evolved into a greyhound sidecar service (gRPC/protobuf) with resilient producers, disk/object backup, parallel and batch consumers, context propagation and custom metrics. Work includes multi‑cluster partitioning, a replication service that routes only to regions with active consumers and a main‑DC producer path for write‑capable zones.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 2:32Kafka-based global data architecture
3. 7:13Scaling Kafka for microservices
4. 10:06Greyhound event processing library
5. 16:46Greyhound features
6. 19:36Event-driven ecosystem with Greyhound
7. 19:57Kafka cluster overload solutions
8. 23:10Cross-data center event consumption
9. 28:04Seamless event production
10. 30:13Next-gen global replication
11. 32:42Summary and open-source
12. 34:06Q&A

" related talks

• 35mPolyglot, Fault-Tolerant Event-Driven Programming with Kafka....Natan Silnitsky · Devoxx Ukraine 2020
• 54mLessons learned from working with 2000 event-driven microservicesNatan Silnitsky · GeeCON 2023
• 30m8 Lessons Learned from using Kafka with 1000 microservicesNatan Silnitsky · Scale By the Bay 2020
• 41mBattle-tested event-driven patterns for your microservicesNatan Silnitsky · Scale By the Bay 2021
• 43mBattle-tested event-driven patterns for your Microservices archit.Natan Silnitsky · Devoxx Ukraine 2020
• 52m10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICESNatan Silnitsky · Scalar 2020
• 46mLessons Learned From Working with 2000 Event-Driven MicroservicesNatan Silnitsky · Build Stuff 2022
• 57mOur DataMesh Journey: data architectures for microservicesCarlos Saona-Vázquez · Spring I/O 2022