How to run Evals at Scale: Thinking beyond Accuracy or Similarity
summary
Argues that evaluations must measure actionable things not just accuracy, showing concrete practices: start with synthetic data then expand to multiple labeled datasets covering flows; use adaptive metrics per application (RAG uses similarity/usefulness, code gen uses functional correctness, agents need trajectory evaluation). Scale by capturing intermediate results, running parallel orchestration and regression suites, and automating frequent runs while balancing human-in-the-loop reviews. Promotes "eval development" like TDD and a strict measure monitor analyze repeat cycle.
AI-written summary · about
related talks
31mBeyond Vibes: Building Evals for Generative AI
19mThe maturity phases of running evals
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
22mCracking the Code: Deciphering Evaluation Essentials for RAG
47mBuild Trustworthy AI Agents powered by Evals
15mEnsure AI Agents Work: Evaluation Frameworks for Scaling Success
41mFrom Lab To Life: Practical AI System Evaluation
06mThe Future of Evals: From LLM as a Judge to Agent as a Judge