Building Trust with LLMs
summary
Compares three concrete evaluation paths: automatic metrics like ROUGE and BLEU and F1 which miss semantics, model‑based scoring using LangChain criteria metrics which is cheap but highly variable, and human evaluation collected via Argilla for domain accuracy. Builds a POC that generates API docs from Python, runs models including OpenAI GPT‑3.5 and others on a Red Hat cluster, and finds generated docs readable but logically weak (demo assigns a low logic score), so human review and tailored benchmarks are required.
AI-written summary · about
chapters (11)
- 0:00Building trust with LLMs
- 1:29Large language models overview
- 3:37Evaluating LLMs for safety
- 5:22Automatic evaluation metrics
- 9:15Model-based evaluation
- 12:04Human evaluation for accuracy
- 16:19LLM benchmarks
- 23:25API docs generation demo
- 27:45Human feedback with Argilla
- 30:14Unified LLM evaluation challenges
- 31:42Closing remarks
related talks
49mNext-Gen Documentation with AI
35mEvaluating LLM systems: the nuts and bolts
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
46mBeyond BLEU & ROUGE — Evaluating LLMs and AI Systems
46mEnhancing Agent Reliability Through Evaluation-Driven Development
28mTrustworthy LLMs: Vibe checks are not all you need
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
32mA Practical Guide to LLM Evaluation