Experimenting with AI-Driven Systems
summary
Ad systems run roughly ~1,000 concurrent experiments by thousands of engineers, so production is constantly changing; a new safety pipeline adds offline inference checks (normalized entropy and resource load tests), a canary rollout, automated per‑experiment business metrics, and a risk score to throttle rollouts. The system blocked hundreds of experiments in six months, monitors model drift with Jensen Shannon divergence, offers LLM-driven debugging before automatic shutdowns, and keeps a manual big‑red‑button for immediate mitigation.
AI-written summary · about
chapters (13)
- 0:00AI-driven experimentation intro
- 1:00Late-night incidents at Meta
- 5:36Default experiments in production
- 13:40Experiment safety initiative
- 15:24Change safety in ML experiments
- 18:13Offline validation and resource estimation
- 19:56Canary rollouts and risk decisions
- 22:42Monitoring experiments in production
- 25:29Automated mitigation and emergency stops
- 26:53People and process changes
- 28:16Lessons from reliability improvements
- 29:23Reliability versus developer velocity
- 30:03High-quality monitoring importance
related talks
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
14mHow leading AI companies 10x experimentation velocity with Statsig
33mEngineering AI Applications: A Conversation with AI Leaders
26mBecoming AI engineers
44mAgentic Observability — The Path to AI Co-SRE
43mTaming the Unpredictable: Reliability in Chaos
09mAI and Chaos Engr: Smarter Failure Testing for Resilient Systems
08mProduction Evals For Agentic AI Systems