Where Have All the Metrics Gone
summary
Explains how RAG systems and LLM agents fail in concrete ways and how to measure each failure: domain failure (authoritative but subtly wrong, only catchable by experts), form failure (invalid JSON/HTML checked with a binary validator or enforced with pyantic and outlines), and consistency failure (measure via average pairwise embedding similarity across many generations). Recommends picking two to four relevant failure modes, using piest for structured evaluations, Likert scoring for expert judgment, and small, repeatable experiments.
AI-written summary · about
chapters (17)
- 0:00Disappearing metrics problem
- 1:33Traditional ML metrics insufficient
- 4:23LLM errors multi-dimensional
- 9:19New failure mode vocabulary
- 10:04Domain failure in generative AI
- 13:55Form failure and structural issues
- 15:43Mode collapse in LLM content
- 18:40Consistency failure across runs
- 20:46Boundary failure and unintended behaviors
- 22:31Temporal failure in evolving contexts
- 25:05Failure planning in agent design
- 26:51Projects as measurable experiments
- 28:56Workflow decomposition for errors
- 32:38Measurement strategies for wrongness
- 34:25Case study agentic client briefing
- 39:00Future of multi-dimensional AI evaluation
- 40:05Closing and Q&A
related talks
29mPutting the data science back into LLM evaluation
11mRAG Evaluation Is Broken! Here's Why (And How to Fix It)
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
22mCracking the Code: Deciphering Evaluation Essentials for RAG
33mThe Silent Crash:Why Your RAG Evaluation Metrics Are Lying to You
18mThe Failures That Don't Crash: MLOps for AI Agents
38mMaking LLMs reliable - A practical framework
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications