How [not] to evaluate your RAG
summary
A production RAG for a fintech support-agent assistant is built on a self-hosted Mixi/nixie search backend, with documents converted to Markdown and a practical GPU-poor contextual chunking strategy that prepends long titles to paragraphs. Embedding experiments compare E5-large against generative models (Gamma 2→3) and show retrieval NDCG gains (~85→89). Evaluation uses automatic context precision/recall, cosine-average context scores, and an LLM judge with 16-page rubric for relevancy; system runs on two GPUs at ~100 tokens/sec and meets pragmatic quality needs.
AI-written summary · about
chapters (6)
- 0:00Introduction and motivation
- 6:22Building a RAG prototype
- 14:34Evaluating retrieval performance
- 24:09Choosing generation models
- 33:19Key lessons learned
- 34:25Q&A session
related talks
22mCracking the Code: Deciphering Evaluation Essentials for RAG
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
31mEvaluating RAG Applications: From Retrieval to Response Quality
22mThe Power of RAG at Scale: A Year of Explosive Growth and Observab...
27mOptimizing RAG Systems in Practice: Challenges and Proven Solutions
11mRAG Evaluation Is Broken! Here's Why (And How to Fix It)
25mAdvanced RAG Systems: Scaling Retrieval-Augmented Generation in Production
56mRAG: from dumb implementation to serious results