Lessons Learned from Our Main Database Migrations
summary
Facebook migrates UDB from compressed InnoDB to MyRocks and cuts storage and server count roughly in half after a nearly yearlong rollout. Migration uses a first-data-load path to bulk-load sorted SSTs, an online copy tool myrocks-hotbackup, and a shadow testing replay of captured writes to validate behavior. Surprises include compaction bugs that leave secondary-index mismatches, tombstones retained after heavy deletes, commit stalls fixed by upgrading Linux to 4.6, and core dump problems resolved by disabling core dumps.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:53Agenda Introduction to Facebook's MySQL and User Database
- 2:45User Database (UDB) at Facebook Storing Social Graph - Massively Sharded
- 10:02MyRocks features Clustered index same as InnoDB Bloom Fiter and Column Family Transactions, including consistency between binlog and Rockste - Faster data loading, deletes and replication Dynamic Options
- 16:12Creating first MyRocks instance without downtime Picking one of the innostave instances, then starting logical dump and restore
- 19:28Shadow traffic tests We have a shadow test framework
- 23:49Stalls on COMMIT Every few minutes, on busy servers, we started observing COMMIT taking 100m a few seconds
- 26:27A few deleted rows re appeared Some of our secondary indexes had extra rows Tumed out to be a bug in RocksDe compactions that in rare cases, on heavy deletions tombstones might not have been handled correctly
- 27:25Debugging root causes How did we find the root cause? Drained the affected MySQL instance for investigation Found last updated timestamps of the extra rows Searched SST files that contained the keys
- 29:35Reproducing Did not reproduce with INSERT Did not reproduce when selecting rows with primary key lookups Sometimes reproduced on range scan
- 30:32Fixing RocksDB and MyRocks Debugged with the reproducible test cases Concluded there were four different bugs
- 31:26Writing core files crashed servers After enabling core files for mysqld process, we started observing more host reboots (unplanned kemel crashes) than before
- 32:46Large virtual memory MyRocks/Rockson built with jemalloc takes very large virtual
related talks
47mLessons Learned from Our Main Database Migrations at Facebook
50mInnoDB to MyRocks Migration in Main MySQL Database
52mRocksDB Performance and Reliability Practics
12mMyRocks at Facebook and a Roadmap
51mMyRocks deep dive and production deployment at Facebook
57mRocksDB
27mMyRocks: best practice at Alibaba - MySQL Storage Engine
29mCharacterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook