Data Consistency at Scale in Meta
summary
Meta claims end-to-end data consistency for large MySQL rollouts (MyRocks and 8.0) by aligning GTIDs, opening consistent snapshots, and computing server-side fingerprints. The tooling uses GTID alignment plus chunked CRC32 fingerprints with a null-bitmask to avoid false negatives, plus pair-mode checks scheduled fleet-wide (full coverage in weeks) and a live replication pre-image check to catch short-lived drifts. Built systems find bit flips, broken backups from extra backup behavior, and surface exact row diffs so operators can investigate and remediate.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:17Example of Correctness Expectation
- 5:28Terminology
- 7:14Implementation - GTID Alignment
- 12:17Implementation - Checksum
- 21:50Implementation - Pinpoint Drift
- 24:01Issues & Tradeoffs
- 29:03Implementation - RBR Consistency Check
- 38:41Implementation - Backup Validation
- 40:26Implementation - Shard Migration
- 42:22Investigation
related talks
49mOnline Database Reliability, Performance, and Consistency Engineering
53mGalera Cluster Data Consistency
39mZero-Downtime Rebalancing and Data Migration of a Mature Multi-Shard Platform
49mMySQL GTID Implementation, Maintenance, and Best Practices - The Short Version
22mMySQL Replication: Latest Developments
46mDatabase Reliability Engineering for MySQL
51mReplication latest features: the community sceptic roundup
48mReducing Risk Before Upgrading MySQL