Hyperscale Kernel Live Patching at Meta

on YouTube ↗28m

" summary

Meta deploys kernel live patching as RPMs with automated health checks and cumulative patch atomic swaps using the replace flag. Applying a KLP takes 1–2 seconds per host and the fleet rollout now typically completes in a week (hours in extreme cases) across millions of servers. Encountered failures include ksoftirqd starvation from insmod CPU hogging (fixed by adding cond_resched and upstreamed), ftrace/BPF IP-modify conflicts (mitigated in newer kernels), and task transition edge cases that require tracing and upstream fixes.

AI-written summary · about

" chapters (11)

1. 0:00Introduction
2. 1:29Full reboot
3. 1:59Pros and cons
4. 3:14KExec
5. 3:56Pros Cons
6. 4:46klp
7. 10:14Ecosystem
8. 16:40Challenges
9. 17:15Bugs
10. 18:01Problem
11. 20:35Tracing

" related talks

• 33mLinux Kernel Live PatchingHaishuang Yan · KubeCon + CloudNativeCon China 2019
• 34mLivepatch Visibility at ScaleBreno Leitao · Linux Plumbers Conference 2023
• 24mKLP for Clang LTO KernelYonghong Song · Linux Plumbers Conference 2023
• 48mAutomated patch management with Ansible and RundeckRico Spiesberger, Andreas Lehr · OSDC 2019
• 15mKVM Live Upgrade with Properly Handling of Passthrough DevicesZhiming Feng · KVM Forum 2020
• 24mContinuously Integrating the Upstream Linux Kernel on HardwareAnders Roxell · foss-north 2019
• 33mHow We Learned to Fix Planes MidflightEvgeniy Kolomeets · HighLoad++ 2017
• 27mAn SRE guide to Linux Kernel upgradesIgnat Korchagin · SREday London 2022