Reliable Data Replication
summary
Shopify builds a partial filtered replication pipeline that reads source MySQL binlogs with Go MySQL Canal, performs parallel backfills with a DB-log style de-duplication algorithm instead of long transactions, writes statements into regional MySQL, and relies on MySQL replication to fan out globally. It enforces correctness with incremental and full-table verification, implements parallel replication right-set algorithm for dependency-aware concurrent writes, and achieves ~5M rows/sec throughput and under 500ms p99 lag while handling online schema changes and continual backfills.
AI-written summary · about
chapters (15)
- 0:00Introduction to data replication
- 1:08Background on Shopify data
- 3:36Problem and requirements explained
- 5:22Current replication at Shopify
- 6:25Why filtered replication
- 7:52Our replication solution overview
- 8:55Handling historical data backfill
- 10:40Evolution of the system
- 11:00Hard problem one correctness
- 12:48Hard problem two schema changes
- 14:32Hard problem three performance
- 17:41Solution summary and impact
- 18:46Downsides and operational complexity
- 19:48Personal learnings and advice
- 20:51Closing and Q&A
related talks
39mZero-Downtime Rebalancing and Data Migration of a Mature Multi-Shard Platform
39mScaling Shopify's Multi-Tenant Architecture across Multiple Datacenters
14mThe State of the Art of MySQL Replication
31m#PostgreSQL to #MySQL: A DBA Perspective
49mMySQL Replication — Advanced Features
22mMySQL Replication: Latest Developments
20mMySQL Binary Logs From Replication Tool to Enterprise Event Platform
29mEverything You Ever Wanted to Know About MariaDB Replication but Were Afraid to Ask