Linux Kernel Building, Testing, and Deployment at Facebook
summary
Facebook moves to an upstream first kernel workflow, building kernels in containers that produce reproducible RPMs on an hourly cadence for every diff, plus official monthly releases; each build is boot-tested on real hardware (not VMs) and run through multiple test suites (autotest, LTP, xfstests, trinity, fuzzers) with results normalized to JSON. Deployment uses 1% canaries and A/B replay tests, four rollout methods, automated escalation instead of mass rollback, and monitoring via kdump and a netconsole daemon.
AI-written summary · about
chapters (28)
- 0:00Intro
- 3:17Building the kernel
- 4:37Previous attempts
- 6:05Automation
- 6:34Existing Infrastructure
- 8:25Build Automation
- 9:09Testing
- 9:44Why is this hard
- 10:50What we do
- 11:34Testing frameworks
- 13:48Systems under test
- 14:16Test health
- 15:08Do I really want the kernel
- 17:15Kernel Release Schedule
- 18:08Grouping the Machine
- 18:44Go Slow
- 19:55The Fleet
- 23:03One Source of Truth
- 23:28Automate
- 27:07Summary
- 28:04Current State
- 28:39More Testing
- 29:49Patching
- 30:22Contributing back
- 31:29Always roll back
- 32:25New log message
- 33:04Cardup issue
- 34:02Build environment issues
related talks
46mLinux Kernel Functional Testing, or How to Build and Test Millions of Linux Kernels
27mKernel Upgrades at
24mContinuously Integrating the Upstream Linux Kernel on Hardware
48mLinux Kernel Developer Panel
23mMoving Fast With Complex...
27mFaster Linux Kernel Testing with Linaro's Open-Source Tools and LKFT Automation
43mWhat's new with CentOS at Facebook
38mkernel: build system outputs and workflows (and how to balance them)