Parallel Asynchronous Replication b/t YDB Database Instances
summary
YDB implements asynchronous cross-cluster replication by streaming tablet change logs into Kafka-like light topics with multiple partitions, hashing each change by primary key so a given row’s updates go to the same partition (hash-by-primary-key). Each tablet acts as a producer with monotonic producer IDs so the partition can tell which records it already saw and dedupe; tablets emit hard-beat records when idle so consumers can advance barriers. Per-partition consumers plus a replication controller form a quantum-based barrier (Calvin-style timestamps) to publish consistent, read-ready snapshots on the target; feature lands in the next major release and is in open source.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:07What is YDB and problem setup
- 7:40Need for async replication
- 10:47Async replication requirements
- 11:08YDB architecture components
- 15:43Replication issues in distributed systems
- 21:27Data log solution
- 23:57Partitioning topic records
- 27:50Handling duplicates and log growth
- 32:33Replicating to target cluster
- 33:36Distributed transactions and consistency
- 41:22Consistent snapshots on replica
- 48:56Summary and future plans
- 50:46Q&A
related talks
30mAn approach to unite tables and persistent queues in one system
29mData Modeling for Distributed SQL
36mRolling Updates: Database Version Migrations with minimum Disruption
01h 27mYugabyteDB YSQL Development
26mxCluster Replication: Discover The Power and Flexibility of Asynchronous Replication
32mArchitectural Flexibility and Geo-Specificity with Cluster-to-Cluster Replication
27mIntroduction to YugabyteDB: Design and Architecture
27mIntroduction to YugabyteDB – Design and Architecture