Measuring Generative AI Solutions
summary
Describes concrete measurement: break ground truth into discrete claims and use an LLM as the judge to score answer correctness, e.g. an insurance-policy question yields two of three claims present → 67% answer correctness. Shows a chatbot built by ingesting Wells Fargo site content that answers and cites sources and handles an age constraint but scores ~52% correctness, and argues deployment requires ~95–99.7% measured correctness before production.
AI-written summary · about
related talks
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
30mIs This Actually Getting Better? A Three-Layer Approach to Measuring AI Products
44mProduction-Grade LLMs & AI Agents
42mRobust Testing Strategies for AI-Driven Applications
24mResponsible AI: Assessing the Truthfulness of AI Model Responses
25mAI vs. AI: How to Test AI Systems Using AI
25mFrom Chaos to Control: Building Enterprise-Grade LLM Agents That Actually Work
15mPractical tactics to build reliable AI apps