Panel: Facebook's Distributed Database Evolution
summary
Facebook starts with 2,000 per-school MySQL copies for about 2M users in 2005, where user IDs encode shards and transfers require scanning every copy; 2006 introduces a networks table and a generalized object/association model to avoid 2,000 schema changes; cache coherence becomes a focus with the Tao single-populate cache pattern and a global pub/sub called Wormhole; RocksDB evolves from LevelDB to optimize LSM on flash and enable large secondary indexes; ZippyDB becomes the multi-tenant distributed KV with SMC2 and shard manager; teams report real incidents — JVM-backed backups corrupting data and a dropped production DB restored within minutes.
AI-written summary · about
chapters (6)
- 0:00Start
- 11:53How Much Ram We Needed on Our Servers Relative to the Amount of Data
- 21:32The Development of Physique Pdb at Facebook
- 35:10Secondary Indexing
- 39:30Shard Manager
- 43:15How Do You Keep the Engineers Interested
related talks
47mLessons Learned from Our Main Database Migrations at Facebook
30mData @Scale Seattle- Muthu Annamalai
57mRocksDB
52mRocksDB Performance and Reliability Practics
51mMyRocks deep dive and production deployment at Facebook
39mLessons Learned from Our Main Database Migrations
43mMulti-tenant HBase Solutions at Facebook
12mFacebook's Tectonic Filesystem: Efficiency from Exascale