Build CDC pipeline using Kafka ecosystem

on YouTube ↗02h 03m

" summary

Shows a hands‑on CDC pipeline built with Docker Compose using MySQL → Kafka Connect → Kafka → BigQuery, including ksqlDB, Schema Registry, Kafka UI, Prometheus/Grafana. Explains Kafka Connect’s six components and that at least two connectors (source + sink) are required; demo creates connectors, streams snapshots and binlog CDC, applies SMT transformers and a Dead Letter Queue, and successfully delivers updates to BigQuery. Claims Kafka handles millions/sec and demonstrates schema registry cuts payload from ~3 KB to 61 B. Notes schema changes must be additive (add columns).

AI-written summary · about

" chapters (13)

1. 0:00CDC pipeline with Kafka
2. 3:05Analytics pipeline challenges
3. 16:41OLTP vs analytics limits
4. 33:35Change Data Capture importance
5. 42:16Kafka ecosystem overview
6. 56:40Kafka Connect with Docker & MySQL
7. 1:06:47Kafka infrastructure health
8. 1:14:36MySQL source connector setup
9. 1:29:24MySQL to Kafka with Avro
10. 1:36:52Analytics isolation with CDC
11. 1:45:46Kafka to BigQuery
12. 1:58:38CDC schema changes handling
13. 2:01:43CDC benefits for teams

" related talks

• 44mNo More Silos: Integrating Databases and Apache KafkaRobin Moffatt · code.talks 2019
• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · LISA18
• 30mAccelerate Modernization with Confluent and YugabyteDBAnanda Bose, Amey Banarse · Distributed SQL Summit 2022
• 39mDeep Dive on Using YugabyteDB's Change Data Capture ConnectorBalachandar Seetharaman · Distributed SQL Summit Asia 2023
• 26mConnecting distributed data with Apache KafkaJacob Bogie, TJ Laher · Azure Cosmos DB Conf 2022
• 03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...F. Ramière, D Petisme · Devoxx France 2021
• 43mContinuous SQL with Kafka and FlinkTimothy J Spann · JCON WORLD ONLINE 2023