Livepatch Visibility at Scale
summary
Meta collects console output from millions of servers using netconsole and adds a kernel version to each message, but livepatches break that mapping. A production hack appends a hotfix suffix via a macro HV_hotfix_version into printk so netconsole carries a netconsole kernel-version tag, but it fails when multiple livepatches exist and UDP drops messages. Crash analysis currently assumes a loaded module equals an applied patch. Proposals: extend utsname or add a netconsole dictionary field or expose a simple /sys/kernel/livepatch active list.
AI-written summary · about
chapters (9)
- 0:00Introduction to Livepatch
- 1:05Livepatch use cases at Meta
- 3:13Livepatch rollout phases
- 6:43Monitoring challenges
- 10:39Visibility of applied patches
- 14:52Proposed solutions
- 20:54Discussion on patch tracking
- 25:35Performance impact of livepatch
- 32:44Transition errors and debugging
related talks
28mHyperscale Kernel Live Patching at Meta
33mLinux Kernel Live Patching
33mHow We Learned to Fix Planes Midflight
14mThe Power of Backports : Business Impact and Automation Strategies
48mAutomated patch management with Ansible and Rundeck
51mEngineering Live Ops on Call of Duty: WWII
30mYou Might Still Need Patches for Denim
42mScan, Patch, VEX - Using Open Source Tools to Manage...