systemd-sysext in Prod: What We Learned Extending /usr Without a Package Manager
summary
Flatcar has been shipping systemd-sysext overlays since 2022 and runs Kubernetes, Docker and containerd as overlay images so the cluster can upgrade Kubernetes independently via sysext + sysupdate. Production use for two and a half years in production surfaces two hard failures: dynamic linking collisions and library path isolation. The team builds tools (Flix and Flatwrap) that rewrite loader paths or bundle libraries to make overlays self-contained; opt-in/opt-out and community bakery workflows work, and comparisons to rpm-ostree clarify when sysext is appropriate.
AI-written summary · about
chapters (10)
- 0:00Introduction to sysext
- 2:46Immutable OS challenges
- 5:32Understanding systemd-sysext
- 7:41OS-dependent vs independent
- 8:37Live demo: managing updates
- 17:52Technical hurdles and solutions
- 20:02Flatcar extension architecture
- 23:25Conceptual models and comparisons
- 25:32Cross-distribution synergy
- 29:16Q&A and concluding remarks
related talks
18mUsing Systemd to Ship Container Runtimes. How?
09mLightning Talk: Shipping Container Runtimes as Systemd-Sysext Images
38mContainer Linux: The Next 10 Years
41mReinventing Container Linux for the Wasm Era (and More) with System Extensions
38msystemd @ Meta: From upstream to production
50mPortable Services Are Ready To Use
31mSolving All the Problems with systemd
01h 05mImmutable and Stateless Systems with systemd