RAGElo: An Elo Rating-based Evaluation Toolkit for RAG
summary
RAGElo is an open-source toolkit that ranks RAG pipeline variants by running pairwise Elo ranking matches where an LLM judges two answers plus their retrieved passages and explains why one wins; it consumes CSVs of queries, retrieved docs, and answers and emits ranked scores. It uses LLMs (example: GPT‑4‑turbo) for relevance labeling, supports synthetic training queries via InPars (10k–100k pairs), and helped move an experimental retrieval stack from Precision@10 ~30% to ~80% after embedding upgrades, fine-tuning, hybrid search and reranking.
AI-written summary · about
chapters (7)
- 0:00RAGElo and RAG intro
- 10:30Why evaluation matters
- 15:29Automated evaluation challenges
- 21:26Elo ratings for evaluation
- 25:27Using RAGElo step-by-step
- 30:07Advanced RAGElo use cases
- 34:44Conclusion and future
related talks
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
33mFrom LLM-as-a-Judge To Human-in-the-Loop: Rethinking Evaluat...
05mopen-rag-eval: RAG Evaluation without "golden" answers
30mEvaluating the evaluator- RAG eval libraries under the loop
31mEvaluating RAG Applications: From Retrieval to Response Quality
11mRAG Evaluation Is Broken! Here's Why (And How to Fix It)
27mOptimizing RAG Systems in Practice: Challenges and Proven Solutions
15mEvaluating RAG Model Performance Metrics, Bias, and Interpretability