Configuration Is (riskier than?) Code

on YouTube ↗32m

" summary

Claim: configuration is actual code and causes many high‑impact outages because a few characters can trigger thousands of actions. Evidence comes from public post‑mortems, a manual review of ~87 SRE newsletter links (only 3 explicitly cite config changes) and an internal Google post‑mortem set where config and binary pushes are similar in frequency (config slightly higher but not statistically significant). Examples include Terraform/Puppet/CFEngine stacks and YAML/JSON lacking debuggers. Mitigations shown: validation, put scripts in VCS, diff generated configs, shift left testing, fuzz/property testing and progressive rollouts.

AI-written summary · about

" chapters (8)

1. 0:00Start
2. 3:23What is configuration?
3. 8:53and a configuration C We want the output of the program to be changed based on some administratively controlled input. This new input is called the configuration
4. 9:29We can change the behaviour without changing the program Given a different configuration C
5. 29:36Property-based Testing and Fuzzing Recall our state space is the Cartesian product of the dimensions of our config C with a possibly large but finite number of values.
6. 31:08Delete code to simplify config Simplicity is hard work, but things to look for: • obsolete config flags never set, or set to defaults. Delete the condition and
7. 31:21Change the power of your config language • Low power languages are less likely to have unexpected side effects inside their own scope. Power reduction allows automation to make more assumptions and optimisations about intent
8. 31:35Configuration is Code, treat it so 1. In theory, configuration should be a high risk of outage 2. Experimentally, not enough data to make strong conclusion 3. Risk mitigations that treat config the same as code work very well a Everything in version control and no out-of-band maintenance

" related talks

• 19mConfig as CodeNilesh Mevada · Agile India 2021
• 30mConfiguration Management PatternsBeau Harrington · RailsConf 2013
• 48mWhy checking your infrastructure-as-code for misconfigurations is not enough – How ...Aakash Shah · Global AppSec US 2021 Virtual
• 20mConfiguration Safety at Scale with AdsJoseph Eley · Systems @Scale Summer 2022
• 01h 19mThe Business of Configuration ManagementMark Burgess · USENIX Configuration Management Summit 2013
• 32mThe Necessity of Configuration and System Management ToolsMarkus Holtermann · PyCon Australia 2015
• 27mCatch Me If You Can: Hunting Misconfigurations Before They...Jagadeesh Devaraj, Marcel Punselie · SREcon25 Europe/Middle East/Africa
• 49mEverything is Terrible: Three Perspectives on Building, Configuring, and Securing SoftwareBill Weiss, Chris Barker, Adrien Thebo · AppSecUSA 2016