Streaming Data Integration for Microservices with Kafka and Kafka Connect
summary
Describes a Kafka-based integration platform built to stream authoritative, real-time stock levels from a master relational database to other systems via API. Covers why Kafka and Kafka Connect were chosen, which connectors were used, the list of source DBs and needed transformations, selection of Tarantool as the target datastore, services that consume/enrich/filter Kafka events and write to Tarantool, and monitoring choices (Dynatrace, Prometheus, Burrow), achieving end-to-end latency of 5–10 seconds.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 2:11Kafka at Leroy Merlin
- 6:12Project goals
- 8:41Kafka Connect for DB
- 10:29Project architecture
- 13:22Expanding Kafka use
- 14:24DB integration volumes
- 16:38Why Kafka Connect
- 19:05Connectors and config
- 23:00Results and benefits
- 24:04Data formats compression
- 25:33Tarantool storage
- 26:58Tarantool testing
- 28:48Why Tarantool solution
- 30:36Monitoring Kafka Connect Tarantool
- 34:14Pitfalls connector issues
- 39:33Project results
- 41:41Future plans
- 42:10Q&A
related talks
59mBuilding Data Intensive Microservices with Apache Kafka
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
21mHow Apache Kafka® Generates Value In Retail With Confluent
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
39mKafka as a Platform: The Ecosystem from the Ground Up
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
41mKafka as a Platform: The Ecosystem from the Ground Up