Configuration Safety at Scale with Ads
summary
Meta runs ~10,000 generated config files used by >100 critical Ads services with ~1,000 engineers and ~400 manual commits per day; historically ~60% of config changes had no testing and two major outages traced to config lands. Engineering adds build-time validations, automated diff-time testing that suggests and launches integration tests (raising coverage from ~20% to ~80% for critical services), and a new consumption-based gradual rollout service that can push changes incrementally and prevent ~99% of bad commits before they hit 100% production.
AI-written summary · about
chapters (10)
- 0:00Config safety at Meta
- 3:39Power and risks of config changes
- 5:24Recent config incidents
- 7:34Why reliability matters
- 9:41Directory-based selection limits
- 12:27Consumption-based selection
- 13:54Gradual config rollout
- 16:22ML for better test plans
- 18:10Speed versus reliability
- 19:13Future config challenges
related talks
32mConfiguration Is (riskier than?) Code
27mCatch Me If You Can: Hunting Misconfigurations Before They...
18mMaking Deployments Safe at Meta
30mConfiguration Management Patterns
25mFast, Safe, and Reliable: The Future of Configuration
31mSaaS release engineering: balancing speed, safety & scale
18mLimiting the Blast Radius: Preventing Outages from Bad Configs with Datacenter-Scale Health Checks
21mScaling Releases: Inside Meta's WWW Release Operations