Data replication patterns 101
summary
RealCo needs millisecond freshness for a hybrid feature with ~two months of data available from an API and limited platform infrastructure, so bootstrap + delta wins: it bootstraps a full CSV snapshot then applies frequent deltas, but it induces production datastore coupling and replay complexity. An event log gives easy replay and true CDC but requires mature infra (Kafka-style) and org support. A daily snapshot suits analytical loads but misses real-time freshness. Idempotency uses upserts or hashed IDs at the destination.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:42What we're going to cover today
- 1:47An example we can use
- 3:16Data use cases
- 6:48Core time projection
- 8:50Coupling in data engineering
- 10:49Data replication patterns
- 11:16Daily Snapshot
- 13:56Bootstrap with Delta
- 19:02Eventlog
- 23:13Summary of the patterns
- 27:11Idempotency
- 28:52RealCo Requirements
- 31:47What is a data product
- 32:29From replication pattern to data product
related talks
20mChange Data Capture in Distributed Systems
45mModern Data Architecture
19mChange data capture with Debezium…and without
21mBuilding Data Integration Services In Real Time
47mRethinking Relational Database CRUD Design Patterns Since We Live in a Big Data World
25mCloud native data replication strategies
38mCDC Without Pain: real-world resilient replication with Debezium and Kafka
39mData Observability and Reliability Engineering in a Real-Time World