Measuring the unmeasurable: How to evaluate generative AI-based systems?
summary
Practical guidance on evaluating generative-AI systems (LLMs) including limitations of classic metrics and automated evaluation via “LLM-as-a-judge” (and its biases), the role of human evaluation, and continuous monitoring to detect drift and side effects. Uses a RAG chatbot as a running example to show how to balance factual accuracy and linguistic creativity and how to implement a reproducible evaluation pipeline covering technical correctness, coherence, and business relevance.
AI-written summary · about
chapters (11)
- 0:00Evaluating generative systems
- 3:12Trust and success challenges
- 5:59Creativity-control paradox
- 8:06Automated evaluation pipeline
- 11:12Operational and generative quality
- 13:38Comprehensive evaluation dataset
- 18:36Good response criteria
- 23:49Evaluation metrics and alignment
- 29:22LLMs as judges
- 34:37Key steps summary
- 39:50Conclusion
related talks
55mLessons Learned From Implementing Generative AI Solutions
26m7 Habits of Highly Effective Generative AI Evaluations
44mUnderstanding the abilities of AI systems: Memorization, generalization, and points in between
12mObserving AI: Why Even LLMs Sometimes Need Therapy
46mGenerative AI: Say Goodbye to Deterministic Testing?
41mAutonomous Quality Engineering: AI Systems That Generate Execute Heal Learn and Govern Quality
41mFrom Lab To Life: Practical AI System Evaluation
42mAdvancements in Evaluating Large Language Model Applications