Lessons Learned From Working with 2000 Event-Driven Microservices

on YouTube ↗46m

" summary

Wix moves 70 billion Kafka messages per day across ~2,300 microservices and documents concrete fixes: make DB updates and event publishes atomic (resilient producer that falls back to S3 with a healer), add entity revision IDs to discard out‑of‑order updates and event transaction‑id deduplication for idempotency, use a Greyhound library for resilient producers/consumers, default compression and S3 references for large payloads, and provide a Kafka control plane to inspect, skip, replay and rebalance partitions for incidents like Black Friday.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 1:48Event-driven architecture
3. 6:00Service communication
4. 12:13Ensuring resilience
5. 16:40Ordering and consistency
6. 22:00Handling large payloads
7. 29:08Monitoring and debugging
8. 35:38Summary and resources
9. 38:25Q&A session

" related talks

• 54mLessons learned from working with 2000 event-driven microservicesNatan Silnitsky · GeeCON 2023
• 30m8 Lessons Learned from using Kafka with 1000 microservicesNatan Silnitsky · Scale By the Bay 2020
• 43mBattle-tested event-driven patterns for your Microservices archit.Natan Silnitsky · Devoxx Ukraine 2020
• 41mBattle-tested event-driven patterns for your microservicesNatan Silnitsky · Scale By the Bay 2021
• 35mPolyglot, Fault-Tolerant Event-Driven Programming with Kafka....Natan Silnitsky · Devoxx Ukraine 2020
• 52m10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICESNatan Silnitsky · Scalar 2020
• 37mKafka Based Global Data Mesh at WixNatan Silnitsky · DevDays Europe 2022
• 44mScaling Horizons Effective Strategies for Wix's Scaling challengesNatan Silnitsky · GeeCON 2024