open-rag-eval: RAG Evaluation without "golden" answers
summary
Claims automated RAG evaluation without golden answers by using LLM judges and open tooling; framework integrates UMBRELA for reference‑free retrieval scoring, AutoNuggetizer for nugget creation plus LLM support checks, a 0–3 chunk score that University of Waterloo research says correlates with humans, and a top 20 nuggets workflow for generation. Connectors for Vectara, LangChain, LlamaIndex ingest 10–1,000 queries, outputs produce citation‑faithfulness and hallucination metrics, a UI visualizes results, and the repo is open source.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:16What is openrageval
- 0:42Openrageval architecture
- 1:33Metrics
- 2:08Umbrella
- 2:51AutoNuggetsizer
- 3:30Citation faithfulness
- 3:43Hallucination detection
- 3:56User interface
- 4:24Conclusion
related talks
31mEvaluating RAG Applications: From Retrieval to Response Quality
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
22mThe Power of RAG at Scale: A Year of Explosive Growth and Observab...
30mEvaluating the evaluator- RAG eval libraries under the loop
39mRAGElo: An Elo Rating-based Evaluation Toolkit for RAG
01h 28mFrom RAG to Riches Building Enterprise Search Systems with LLMs
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
22mCracking the Code: Deciphering Evaluation Essentials for RAG