The Future of Evals
summary
Braintrust reports the average org runs 13 evals a day and some customers run over 3,000 evals a day; eval work is currently manual but Loop automatically optimizes prompts, datasets, and scorers inside Braintrust, using Claude 4 by default and supporting OpenAI, Gemini, or custom LLMs. Claude 4 is claimed to be almost six times better than the prior leader. Loop is behind a feature flag, shows side‑by‑side edits, and has an optional “auto‑optimize” toggle.
AI-written summary · about
chapters (5)
- 0:00Introduction to AI Engineer World's Fair
- 0:15Speaker Introduction: Ankur Goyal, CEO of Braintrust
- 0:30Increasing Adoption of Eval
- 1:58Introducing Loop
- 4:09Call to Action: Try Loop and Join the Team
related talks
06mThe Future of Evals: From LLM as a Judge to Agent as a Judge
43mEvaluation is all you need
13mHow to evaluate your Evals?
31mBeyond Vibes: Building Evals for Generative AI
27mThe Science and Practice of Open and Scalable LLM Evaluations
46mEnhancing Agent Reliability Through Evaluation-Driven Development
47mBuild Trustworthy AI Agents powered by Evals
32mEvaluating multi-turn conversations