Putting the data science back into LLM evaluation
summary
Argues for reproducible, metric-driven LLM validation and demonstrates an open-source Django web app that compares models prompt-by-prompt, logs generation config, and surfaces automated checks (cosine similarity to expected output, JSON-parsability, PII leakage, repetition, length). Recommends combining classic metrics like precision/recall or perplexity with custom measures (readability Flesch–Kincaid, schema validation) and human blind preferential voting to catch factual errors and safety issues while making model progress visible and repeatable.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:28LLM risks
- 4:43LLM evaluation problems
- 7:17Data science validation
- 11:39LM evaluation guidelines
- 17:43Parland for LLM evaluation
- 23:06Future of LLM evaluation
- 25:59Q&A
related talks
14mTailor-Made LLM Evaluations: How to Create Custom Evaluations for Your LLM
14mImproving Large Language Model Performance
41mWhere Have All the Metrics Gone
34mBuilding Trust with LLMs
27mThe Science and Practice of Open and Scalable LLM Evaluations
13mOptimizing and Benchmarking Open-Source LLMs
42mAdvancements in Evaluating Large Language Model Applications
32mStrategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)