Evaluating AI Search: A Practical Framework for Augmented AI Systems
summary
Introduces an open-source dynamic eval agent built with Langraph that generates web-grounded Q&A, refreshes datasets continuously, and logs runs in lengthmiss; uses aggregated results from multiple real-time AI search providers. Benchmarks six providers on a 1,000-row dynamic set versus static SimpleQA and finds dynamic correctness substantially lower and provider rankings change (one provider flips from worst to best). Shows reference-free metrics (answer completeness, document relevance, hallucination) correlate with dynamic accuracy (completeness vs accuracy r≈0.94), reveals only three providers return grounding docs, and surfaces an unexpected tradeoff: higher document relevance can coincide with higher hallucination.
AI-written summary · about
chapters (8)
- 0:00Introduction to AI search evaluation
- 1:28Real-world challenges in AI search
- 3:35Static vs dynamic evaluation data
- 6:48Building dynamic evaluation sets
- 10:03Reference-free benchmarking methods
- 13:58Evaluating without ground truth
- 18:39Interpreting evaluation results
- 19:43Future of self-improving AI agents
related talks
25mWho Let the Bots Out? A Guide to Evaluating AI Agents
41mAI is here – time to throw away our search engines?
43mHarnessing AI to strengthen trustworthy information
57mEvaluating AI Agents in production with Python
47mBuilding and Testing AI-Agent Powered LLM Applications: A Live Demonstration
41mSurviving and Thriving with AI in QA
01h 00mLLMs gone wild
30mIs This Actually Getting Better? A Three-Layer Approach to Measuring AI Products