Production as an Experiment Lab
summary
Argues that production becomes a controlled field experiment using feature flags, canaries and A/B testing combined with distributed tracing, metrics and fault injection to generate valid data about real user impact. Describes a correct pattern of edge routing and deterministic bucketing and a wrong pattern of DB flag switching that doubled index size and skewed results. Gives a case where routing a subset to a slower pipeline for months saves about $2–3M/year with no revenue hit. Presents error budgets as an opportunity to explore failure space and run multiarmed bandit style experiments.
AI-written summary · about
chapters (16)
- 0:00Start
- 6:02SOLUTION TO COMPLEXITY IS NOT SIMPLICITY
- 6:21PREMISE COMPLEXITY AND FAILURE GO HAND IN HAND
- 9:45LETS REVISIT CHANGE SPECIFICALLY IN CONTEXT OF MEASUREMENT
- 11:50LETS START BY USING CONCEPTS TO ESTABLISH A VOCABULARY FOR THINKING ABOUT INFRASTRUCTURE EXPERIMENTATION
- 12:15TEST IN PROD PROGRESSIVE DELIVERY ERROR BUDGETS
- 15:14PROGRESSIVE DELIVERY IS CONTINUOUS DELIVERY WITH FINE- GRAINED CONTROL OVER THE BLAST RADIUS.
- 16:47FROM CORRECTNESS TO SAFETY
- 18:23NOW WHAT?
- 20:12ERROR BUDGETS OPPORTUNITY FOR LEARNING
- 20:41IT IS NOT YOUR JOB TO CREATE INFINITELY RELIABLE SOFTWARE
- 21:24PERFORMANCE COST CAPACITY FAULT INJECTION NEW ARCHITECTURES
- 23:18WHAT IS THE DIFFERENCE BETWEEN A TEST AND AN EXPERIMENT?
- 24:49WHERE HAVE WE HEARD ABOUT EXPERIMENTS IN INFRASTRUCTURE ALREADY?
- 25:16EXPERIMENTS AREN'T JUST CHANGING STUFF
- 28:00HOW TO DO IT WRONG
