Building Fast ClickHouse Analytics for MySQL Transaction Data Using Kubernetes
summary
A concrete pipeline is implemented: Debezium -> Kafka/Redpanda -> a Kafka Connect sync connector -> ClickHouse on Kubernetes, with Debezium Kafka ClickHouse as the core pieces. The connector batches records into configurable blocks, writes offsets into ClickHouse to achieve exactly-once semantics, uses JDBC bulk inserts, supports Avro + schema registry, and handles updates/deletes via collapsing/replacing merge trees. The proof-of-concept runs on Kubernetes, shows multi-table parallelism and transaction-aware writes, and sustains the quoted multi-million-rows-per-second class throughput in demos.
AI-written summary · about
chapters (11)
- 0:00Start
- 6:23Architecture
- 14:50Kafka Engine
- 15:56Objectives
- 16:57Transaction Support
- 19:18Kafka's Connect Sdk
- 23:34Parallelizing Aspect of Kafka Consumers
- 34:52Updates and Deletes
- 36:26Collapsing Merge Tree
- 43:40Add New Connectors
- 54:37Auto Scaling
related talks
42mStreamlining Real-time Data Analytics: Integrating ClickHouse with Apache Kafka
42mBuilding an Analytic Extension to MySQL with ClickHouse® and Open Source
41mData warehouse and Kubernetes: Lessons from ClickHouse Operator
17mBuilding Real-Time Data Lake with ClickHouse
42mClickHouse: what is behind the fastest columnar database
34mClickHouse® - What is Behind the Fastest Columnar Database?
30mReplicating Data from MySQL to ClickHouse
30mBuilding For Fast