10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICES
summary
Wix runs ~1,400 Scala microservices on Kafka ingesting ~1 billion messages/day and built an internal library called Greyhound to standardize producers/consumers, hide boilerplate and expose features like automatic commits, re-wrote Greyhound with ZIO for lightweight fibers and parallelism, and a testkit that spins up embedded Kafka+Zookeeper. Retry is implemented via retry topics (single-partition tradeoffs) and dashboards feed Prometheus alerts. Polyglot Node.js services use a sidecar for Node.js proxy to Greyhound, and operations guard brokers at ~4,000 partitions and clusters at ~200,000 partitions.
AI-written summary · about
chapters (19)
- 0:00Start
- 0:44Kafka Introduction
- 2:46Wix Setup
- 4:26Lesson Number One about Common Infrastructure
- 4:55Greyhound Library
- 6:24Simpler Consumer Api
- 9:09Parallel Consumption
- 11:28Error Handling
- 14:43Processing out of Order
- 19:03Yahoo Kafka Manager
- 19:58Live Topic Stream
- 22:16Documentation
- 23:54Discover Topics and Message Structures
- 26:58Monitoring
- 29:58Proactive Worker Maintenance Approach
- 31:52Lesson Number Seven about Cross Dc Producing
- 41:48Event-Driven End To End
- 42:26Use Case
- 47:59Sweet Spot of Centralization
related talks
30m8 Lessons Learned from using Kafka with 1000 microservices
35mPolyglot, Fault-Tolerant Event-Driven Programming with Kafka....
54mLessons learned from working with 2000 event-driven microservices
46mLessons Learned From Working with 2000 Event-Driven Microservices
29mGreyhound - Powerful Pure Functional Kafka library
37mKafka Based Global Data Mesh at Wix
43mGreyhound - Powerful Pure Functional Kafka library
39m10 years of Kafka. Tips, tricks and lessons learned