Evaluating LLM Frameworks
summary
An engineer from CM.com describes a Python evaluation framework for comparing LLMs and prompts for a production Retrieval-Augmented Generation (RAG) chatbot used by over 50 clients. The talk covers RAG principles, concrete evaluation metrics and scenarios used to judge model swaps (e.g., ChatGPT vs Gemini vs Llama) and prompt optimizations, and explains how those objective measures guide product decisions to maintain correctness and appropriateness.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:05LLM frameworks overview
- 4:35LLM challenges and hallucinations
- 8:27Retrieval Augmented Generation explained
- 13:00Model-agnostic RAG frameworks
- 14:22Evaluating LLM performance
- 17:08Faithfulness and similarity metrics
- 21:00Automated evaluation setup
- 22:47Key takeaways
- 24:12Q&A: Cost vs performance
- 26:37Q&A: Ground truth challenges
- 28:47Q&A: Filtering and safety
- 30:31Q&A: Turing test agreement
- 31:38Q&A: Prompt engineering
related talks
24mHow to Keep Your LLM Chatbots Real - A Metrics Survival Guide
01h 34mHow to Construct and Assess Your First LLM Based RAG System
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
35mEvaluating LLM systems: the nuts and bolts
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
31mBeyond LLMs and Maturing RAG use cases
10mModel innovation & evaluation with model partners Q&A
01h 39mMastering LLMs: From Prompt Engineering to Agentic AI