Kernel Live Patching at Scale
summary
Facebook uses kernel live patching (KLP) with ftrace IP-modify to push cumulative fixes as RPMs across millions of servers, rolling a new cumulative KLP module in about a week (whole fleet possible in hours). Typical KLP modules are under ~1 MB and replace tens of functions (rarely >100). Known issues: KLP causes transition failures 10–100x in edge cases (long-running workqueue tasks, tracing conflicts); fixes include adding cond_resched and upstream patches.
AI-written summary · about
chapters (19)
- 0:00Introduction and team
- 0:43Kernel fixes and reboots
- 3:10What is live patching
- 6:59Live patching ecosystem
- 8:07Why use live patching at scale
- 10:14Enforcing homogeneous configuration
- 13:04Results and benefits of live patching
- 15:12Bugs and issues encountered
- 16:12Bug: missed I/O and TCP retransmits
- 18:40Live patching and tracing tools
- 22:12Live patch transition failures
- 25:23Ongoing and future work
- 27:50Toolchain improvements
- 29:36Q&A: Assembly file live patching
- 32:52Q&A: Live patch module size
- 35:18Q&A: Link time optimizations
- 36:41Q&A: Functions per live patch
- 38:51Q&A: Kernel version support
- 39:52Q&A: Services affected by PGO
related talks
18mKernel Live Patching: Mitigating CVEs With Zero Downtime
39mLet's meet the kernel livepatch feature
47mAdaptive Kernel Live Patching: An Open Collaborative Effort...
27mAdaptive Android Kernel Live Patching
39mCloudLinux runtime kernel update for ARM growth difficulties
35mKeeping the linux kernel secure across 10+ RHEL releases
26mEngineering for the Long Haul: Operating Open Source at Scale
09mLinux Kernel Bug Fixing Project