Ship Reliable RAG: Evals That Grow with Your Retrieval Complexity
summary
An AI-assisted ticket resolver is built and evaluated across four concrete levels: simple lexical search with BM25 (hit rate ~20%), semantic embeddings (recall rises to ~40%), hybrid retrieval with filtered logs using Drain3 (filtering 50% of log noise loses only ~10 critical error strings and 54 tickets remain resolvable), and hybrid plus reranking where Weighted RRF (weights 1,5,2) and context-window tuning are chosen by the eval suite rather than intuition.
AI-written summary · about
chapters (12)
- 0:00Reliable RAG introduction
- 1:12What is RAG
- 5:48Retrieval complexity levels
- 8:39Evaluating AI systems
- 10:30Lexical retrieval failure
- 11:35Semantic retrieval with embeddings
- 16:19Hybrid retrieval with logs
- 19:10RAG evaluation metrics
- 21:19Reranking strategies
- 24:56Five RAG lessons
- 28:11Balancing automation
- 29:55Closing
related talks
27mOptimizing RAG Systems in Practice: Challenges and Proven Solutions
33mThe Silent Crash:Why Your RAG Evaluation Metrics Are Lying to You
13mThe State of AI Powered Search and Retrieval
59mThe Evolution of RAG
22mThe Power of RAG at Scale: A Year of Explosive Growth and Observab...
49mRAG: from dumb implementation to serious results
11mRAG Evaluation Is Broken! Here's Why (And How to Fix It)
56mRAG: from dumb implementation to serious results