Kernel Upgrades at
summary
Facebook automates kernel upgrades for tens of thousands of database hosts using a fleet of Python state-machine jobs that call kernelctl install kernel X, enforce per-cluster parallelism and canary shards, expose a global kill switch and per-host maintenance handlers for drain/undrain, and record time‑series metrics and upgrade rates. The system converges service configs to desired kernel versions, uncovered service discovery cache bugs and dependent-service rate limits, and upgrades at Facebook scale without disrupting production.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:40Context
- 2:23Why
- 5:46Goals
- 6:48Orchestration
- 7:44Continuous Deployment
- 8:24Solid Monitoring
- 9:48Collaborative Framework
- 11:52State Machine
- 15:44Failures
- 16:04Language specification
- 17:13Constraints
- 18:22Example config
- 19:37Production issues
- 21:31Time series graph
- 22:22Time series database
- 23:05Takeaway
- 23:50Questions
related talks
36mLinux Kernel Building, Testing, and Deployment at Facebook
53mAn engineer's guide to Linux Kernel upgrades
43mWhat's new with CentOS at Facebook
50mUpgrading CentOS on the Facebook fleet
27mAn SRE guide to Linux Kernel upgrades
36mCentOS Stream at Facebook
34mAn SRE Guide to Linux Kernel Upgrades
34mAn SRE Guide to Linux Kernel Upgrades