Validating AI Outputs at Live Events
summary
Built an autonomous validation layer called LLM Judge that combines chain-of-thought prompting, theory-of-mind memory from historical editor edits, and a human-in-the-loop approval loop to tighten content pipelines for live events. Deployed across Wimbledon, the Masters, the Grammys and ESPN NFL Fantasy, it runs 20 iteration edit loops, evaluates outputs with factualness, novelty, relevance and repetitiveness metrics, serves 500,000+ live users, sustains 1,100+ RPS across 280+ models and yields ~30% better summary accuracy and fewer hallucinations.
AI-written summary · about
chapters (6)
- 0:00Introduction to AI validation
- 1:54Challenges with AI content
- 7:33Chain of thought prompting
- 11:25Content evaluation methods
- 16:06Editor correction and convergence
- 20:45Iterative improvement process
related talks
32mEvaluating multi-turn conversations
46mEnhancing Agent Reliability Through Evaluation-Driven Development
25mAI vs. AI: How to Test AI Systems Using AI
43mAI Development Lifecycle: Learnings of What Changed with LLMs
26mReducing Hallucinations and Evaluating LLMs for Production
13mNew Challenges in Software Quality Brought by GenAI and LLMs
01h 44mHow to build world-class AI products
19mBuilding AI Products That Actually Work