Evolution of Development Priorities in Key-value Stores Serving Large-scale Applications
summary
RocksDB evolves over eight years (forked from LevelDB 2012, open‑sourced 2013, production by 2015–16) and shifts optimization goals from reducing write amplification to improving space efficiency, then to cutting CPU cost and enabling aggregated storage (separate compute/storage with network‑side compactions). Implementations include full file checksum metadata, key‑value checksums and handoff checks to detect SSD, copy, and CPU bit‑flip corruption earlier. Work adds tooling for remote compactions, tiered storage and secondary instances; these changes are driven by production failure cases and measured bottlenecks.
AI-written summary · about
chapters (11)
- 0:00Intro
- 2:44Focusing on Write Amplification in Engine Layer
- 4:51It's hard to balance CPU and SSD
- 5:12RocksDB on Disaggregated Storage
- 6:12Recap Performance Optimization Targets
- 6:35Typical Way of Using RocksDB
- 7:20Minimum Integrity Check: Block Checksum
- 8:10Delayed corruption detection risks data loss
- 8:43Keep Full SST File Checksums
- 9:14CPU corruption is silent and can cause data loss
- 10:19Recap Lessons on Integrity Checking
related talks
57mRocksDB
51mSix New Important RocksDB Features And Planned Works
52mRocksDB Performance and Reliability Practics
24mPerformance tuning in BlueStore & RocksDB
29mCharacterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook
46mBlueStore V2 - an Evolutionary Step Forward
01h 02mRonDB: A Key-Value Store with SQL Capabilities and LATS Properties
30mData @Scale Seattle- Muthu Annamalai