Experimentation at Microsoft
summary
Microsoft runs A/B testing and progressive rollouts across Azure and Bing using feature flags and monitoring like Prometheus metrics, driving automated canary deploys that roll to 1/10/50% of users. A concrete win: changing button placement in the Azure create flow reduces completion time and cuts abandonment, measured end-to-end across frontend and backend. Tooling prevents bad rollouts by auto‑stopping on metric regressions, but challenges remain: thousands of flags to manage, accidental flips, and ensuring truly random assignment and correct metric capture.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:18What is AP experimentation
- 2:40Multivariate experiment
- 4:18Measuring changes
- 5:30B testing
- 7:05Prometheus
- 8:20Challenges
- 11:08Customer Obsession
- 13:00Feature Flags
- 14:20Culture around Feature Flags
- 15:20Favorite part about working at Microsoft
- 16:50First impression of Microsoft
- 17:38How do you feel about your deal at Microsoft
- 20:03Should you falsify the data
- 22:26When to remove future flag
- 24:06AP testing strategies for infrastructure
- 25:48Inhouse vs global vendors
- 27:50How involved are they in making decisions
- 30:07How do they make decisions
- 32:49Wallers favorite feature
related talks
16mBuilding blocks for using Azure-native experimentation
19mThink Prod Up Front: Shifting Left on Production Excellence
15mA/B Testing at Scale
24mExperimentation Programs at Scale: Lessons Learned from Top Companies
33mA/B testing: performative experimentation at scale
22mFear of the computer
46mLeverage Azure Testing Services to build high quality applications
21mI Don’t Always Test My Code, but When I Do It’s in Production