Microservices with Kafka

on YouTube ↗54m

" summary

Chooses Kafka as the event bus and builds lightweight producer factory and consumer factory wrappers that add serializers, filtering, transformers, forward/dispatch and a Kafka appender for ELK logging. Benchmarks show one producer handling ~2.7M msgs/sec at 500 bytes and ~9.1M msgs/sec at 100 bytes; logging throughput is ~224k msgs/sec for ~2KB JSON records. Adds a replay capability (retain and reprocess topics) and multi-datacenter cluster setup for durability, and rolls this CFA++ platform into production across countries for banking use.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 20:43Scalability
3. 24:04Near Real-Time Monitoring
4. 25:26The Producer Factory
5. 25:58Producer Factory
6. 30:27Filtering
7. 31:51Transformer
8. 34:18Micro Benchmark
9. 36:05Logging

" related talks

• 33mMicroservices, Events, and Breaking the Data Monolith with KafkaBen Stopford, Cornelia Davis · Cloud Foundry Summit Europe 2019
• 45mReliable Toolkit for Event-Driven Microservice ArchitectureYurii Barabash · Software Architecture fwdays'24
• 01h 00mKafka for .NET DevelopersIan Cooper · NDC Oslo 2024
• 01h 02mKafka for .NET DevelopersIan Cooper · NDC Sydney 2024
• 30m8 Lessons Learned from using Kafka with 1000 microservicesNatan Silnitsky · Scale By the Bay 2020
• 52m10 LESSONS LEARNED FROM USING KAFKA IN 1000 SCALA MICROSERVICESNatan Silnitsky · Scalar 2020
• 33mKafka Microservices with Kafka plus Akka and Kafka StreamsBoris Lublinsky · Scala Days Berlin 2018
• 31mEvent driven communication with Kafka on KubernetesRaihan Khan · KCD Dhaka 2024