No More Silos: Integrating Databases and Apache Kafka

on YouTube ↗44m

" summary

Shows concrete integrations: uses Kafka Connect (JDBC source and binlog connectors) to stream MySQL into Kafka, demonstrates a snapshot then continuous capture, and runs a live join with ksql to enrich incoming rating events with customer data. Compares query-based CDC (simple polling, misses deletes and intermediate updates, adds DB load) with log-based CDC (binlog captures before/after, insert/update/delete), uses Schema Registry and Avro/Protobuf for schemas, and the demos work end-to-end.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 1:25Why use Apache Kafka
3. 4:34Streams of data
4. 5:20Events
5. 14:08Change Data Capture
6. 15:25Transaction Log
7. 15:44What is Transaction Log
8. 23:58Querybased Capture
9. 30:37Sequel
10. 37:25Why PreService
11. 38:25Try it out
12. 38:38References

" related talks

• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · LISA18
• 02h 03mBuild CDC pipeline using Kafka ecosystemKarim Tawfik · DSC MENA 2024
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020
• 42mFrom zero to hero with Apache Kafka's connect APIRobin Moffatt · Berlin Buzzwords 2019
• 01h 07mKafka Streams and KSQLDBDaniel Hinojosa · Great International Developer Summit 2024
• 26mConnecting distributed data with Apache KafkaJacob Bogie, TJ Laher · Azure Cosmos DB Conf 2022
• 45mFrom Zero to Hero with Kafka ConnectRobin Moffatt · Voxxed Days Microservices 2019