Evaluating RAG Applications: From Retrieval to Response Quality
summary
Demo shows evaluating RAG apps end-to-end with RAGAS: retrieve snippets from embeddings and a vector DB, measure context precision and context recall plus faithfulness and answer relevancy, and use an LLM as judge (LLM-as-evaluator). The workflow auto-generates test sets from a knowledge graph, runs assertions (example threshold 0.9) inside pytest, and exports results for CI/CD (CI/CD pytest). The restaurant-finder demo proves retrieval errors drive hallucinations and that fixing retriever quality is the highest ROI.
AI-written summary · about
chapters (9)
- 0:00Evaluating RAG apps
- 3:12Why RAG is needed
- 5:58Testing vs evaluation in LLMs
- 7:23RAGAS introduction
- 9:11Key RAG metrics
- 12:41Demo: RAGAS with restaurant app
- 18:46RAGAS in CI/CD
- 21:15Test data with knowledge graphs
- 24:26Q&A on RAG challenges
related talks
30mEvaluating the evaluator- RAG eval libraries under the loop
15mEvaluating RAG Model Performance Metrics, Bias, and Interpretability
42mTo RAG or not to RAG
27mOptimizing RAG Systems in Practice: Challenges and Proven Solutions
05mopen-rag-eval: RAG Evaluation without "golden" answers
31mAdvanced Enterprise RAG Systems
40mHow [not] to evaluate your RAG
01h 28mFrom RAG to Riches Building Enterprise Search Systems with LLMs