RAG Evaluation Is Broken! Here's Why (And How to Fix It)
summary
Claim: current RAG benchmarks reward brittle systems that score high but fail on real tasks. Evidence: a 22‑document FIFA corpus yields only 5% and 11% correct with common LangChain/LlamaIndex pipelines. Tools tried: clustering, embeddings, reranking, plus a constructed SQL‑backed schema ingest that converts docs to structured rows and uses text‑to‑SQL for counting/aggregation queries. Result: the structured approach fixes many aggregate questions but introduces normalization and ambiguity problems (host country variants, joint hosts) and isn’t universal.
AI-written summary · about
chapters (7)
- 0:00Broken RAG evaluation
- 1:03Flaws in RAG benchmarks
- 3:10Local question benchmark limits
- 4:58Complex aggregative questions
- 6:28Structured RAG for relational data
- 9:00Normalization and ambiguity challenges
- 10:03Key takeaways
related talks
19mImproving RAG Systems - Key Challenges and Solutions
27mOptimizing RAG Systems in Practice: Challenges and Proven Solutions
56mRAG: from dumb implementation to serious results
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
31mEvaluating RAG Applications: From Retrieval to Response Quality
33mThe Silent Crash:Why Your RAG Evaluation Metrics Are Lying to You
22mThe Power of RAG at Scale: A Year of Explosive Growth and Observab...
15mEvaluating RAG Model Performance Metrics, Bias, and Interpretability