Reliably Changing Configuration @ Scale
summary
Meta uses configurator and an automated Auto Canary pipeline that runs structured canary specs to test config changes on real workloads; tooling auto-generates specs and enrollment, while Canary spec doctor continuously measures flakiness by running specs on a never-consumed config. Results: Auto Canary coverage doubles during the push, average individual canary failure rate drops >3x, overall flakiness falls ~60%, and over 80% of services that could trigger a large outage are now protected.
AI-written summary · about
chapters (7)
- 0:00Start
- 1:52Meet Configerator
- 4:26Automated Testing - Canary Specs
- 7:14Config Consumption Patterns
- 7:35Enhanced Enrollment Strategy
- 13:26Taming The Multiple Comparisons Problem
- 16:22Introducing Canary Spec Doctor
related talks
27mCatch Me If You Can: Hunting Misconfigurations Before They...
16mConveyor: One-Tool-Fits-All Continuous Software Deployment
25mOpening Keynote
38mChaos Engineering
33mCI/CD for Autonomous Drive and other complex systems - What we have learned about...
43mQuit Simplifying!
45mConfiguration management for SaaS/BPaaS solutions
27mBuilding a Better Software Supply Chain