rebuilddb at Facebook scale

on YouTube ↗15m

" summary

Facebook-scale CentOS 7 fleet runs Chef every 30 minutes with about 200 engineers constantly installing packages, causing rpm/yum to be killed mid-transaction, produce zero-byte files, duplicate packages and missing shared libs across hundreds or thousands of hosts. Aggressive automatic rebuilds of rpmdb create hundreds of locked rpm processes, so a custom tool dcrpm rewritten in Python runs on every Chef run to detect and repair RPM DBs. New backends nDB and LMDB are backported and tested at limited scale and prove far more resilient than BDB, informing a possible move to DNF.

AI-written summary · about

" chapters (6)

1. 0:00Introduction and context
2. 2:16Chef deployment at Facebook
3. 6:48RPM database issues at scale
4. 8:58Introducing DC RPM
5. 11:34New RPM database backends
6. 13:47Future plans with DNF

" related talks

• 43mWhat's new with CentOS at FacebookDavide Cavalca, Naomi Reeves · CentOS Dojo Brussels 2020
• 41mSpeeding up DNF and RPM using Copy on WriteMatthew Almond · CentOS Dojo FOSDEM 2021
• 36mCentOS Stream at FacebookDavide Cavalca · DevConf.CZ 2021
• 36mLinux Kernel Building, Testing, and Deployment at FacebookYannick Brosseau, Phillip Duncan · SREcon16 Europe
• 54mStill packaging like it's 1999?Florian Festi · DevConf.CZ 2020
• 50mUpgrading CentOS on the Facebook fleetDavide Cavalca · DevConf.CZ 2020
• 27mKernel Upgrades atPradeep Nayak, Udupi Kadbet · SREcon18 Europe/Middle East/Africa
• 20mFuture of the DNF package management stackJaroslav Mracek, Daniel Mach · DevConf.CZ 2019