Practical LLM Evals: Down-to-Earth Tooling for Everyone
summary
A hands-on runthrough of tooling and a concrete experiment that actually measures outcomes not buzzwords covering local LLMs via Ollama/Open WebUI and models like Qwen Phi Gemma Llama plus agentic tools and Copilot the presenter automates evals with CSVs JS Axios calls to a local Llama API and compares scores against humans finding a strong trend with Spearman’s ρ > 0.75 shows runtime and resource tradeoffs and documents failures so you get reproducible prompts code and measurable results to judge for yourself
AI-written summary · about
chapters (16)
- 0:00Practical LLM evals intro
- 2:06Speaker background
- 4:56Small experiments to fight AI overwhelm
- 7:30Retrieval chatbot POC
- 9:57Automating applicant evals
- 13:51Testing LLM judge system
- 15:59AI vs human eval results
- 17:49Performance and hardware needs
- 18:31LLM-as-judge and Ragas
- 21:41Apollo chatbot UI
- 22:45Control tests for chatbot quality
- 26:15Chatbot performance dashboards
- 27:39Error analysis and edge cases
- 30:56Evals beyond AI
- 34:33Explore quality evaluation
- 35:21Q&A on bias and trust
related talks
38mMaking LLMs reliable - A practical framework
32mA Practical Guide to LLM Evaluation
35mEvaluating LLM systems: the nuts and bolts
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
25mEvaluating An LLM's Ability To Code
32mStrategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
28mTrustworthy LLMs: Vibe checks are not all you need