Database Native Support for CDC Streaming Pipeline
summary
Most Cassandra CDC pipelines read commit logs, deduplicate across replicas (replication factor 3 producing up to nine replicas), reconstruct full before/after row images, and publish to Kafka or Kinesis; examples use Redis Bloom filter caching, Kafka + Flink for merging, and custom connectors. Proposal replaces per-node log push with a leader-based replica that serializes updates at source to deliver exactly once and ordered per partition in normal operation, at the cost of added coordinator bandwidth, potential latency impact, and a leader election complexity.
AI-written summary · about
chapters (15)
- 0:00CDC streaming pipeline intro
- 1:03What is CDC and why it matters
- 2:07CDC use cases and examples
- 3:34Cassandra CDC processing steps
- 5:24Walmart and Yelp CDC implementations
- 6:53Challenges with current CDC
- 8:40Ideal CDC traits
- 10:03Old and new images in CDC
- 11:50CDC limitations today
- 12:56Coordinator-based CDC proposal
- 14:58Synchronous CDC push impact
- 16:46Leader-based CDC in Cassandra
- 18:53Replica approach and leader election
- 21:27Q&A on leader election
- 22:52Current CDC support and roadmap
related talks
24mCDC: From Zero to Hero
41mStreaming your database - Easier said than done?
32mA Change-Data-Capture use-case: designing an evergreen cache
30mChange Data Streaming Patterns in Distributed Systems
43mLoading into Kafka from a Database: With and Without CDC
55mSolving Classical Data Analytic Task by Using Modern Distributed Databases
27mNo More Silos: Integrating Databases and Apache Kafka
43mBack to fundamentals: from databases to event-driven platforms