Evaluating the evaluator- RAG eval libraries under the loop
summary
An experiment runs 200 RAG queries with embeddings stored in chrom ADB and compares evaluator libraries like RAGAS and TruLens across four metrics (answer relevancy, faithfulness, context precision, context recall). Results show 48% zero relevant chunks in retrieval, frequent null metrics from empty or rate‑limited replies, and model disagreement except for a tight pair where GPT-4 and GPT-3.5 correlate 99%. Implementation hits rate limits, empty replies from Lama models, and non‑deterministic evaluator outputs that raise cost and reliability concerns.
AI-written summary · about
chapters (10)
- 0:00RAG evaluation challenges
- 1:08Why RAG evaluation matters
- 3:41RAG evaluation libraries
- 4:23RAG evaluation metrics
- 9:23Experiment design
- 12:14LLM evaluator results
- 18:54Evaluator disagreements
- 22:49Practical RAG evaluation
- 25:01Key takeaways
- 26:27Q&A
related talks
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
31mEvaluating RAG Applications: From Retrieval to Response Quality
15mEvaluating RAG Model Performance Metrics, Bias, and Interpretability
33mThe Silent Crash:Why Your RAG Evaluation Metrics Are Lying to You
05mopen-rag-eval: RAG Evaluation without "golden" answers
11mRAG Evaluation Is Broken! Here's Why (And How to Fix It)
27mOptimizing RAG Systems in Practice: Challenges and Proven Solutions
39mRAGElo: An Elo Rating-based Evaluation Toolkit for RAG