2025 is the Year of Evals! Just like 2024, and 2023, and …
summary
Claim: enterprise evaluation becomes urgent in 2025 because models move from scoring to acting autonomously, budgets unlock after ChatGPT on Nov 30, 2022, and CEOs/CFOs/CISOs demand measurable ROI and risk numbers. Evidence: timeline of adoption — 2023: GenAI adoption, 2024: first production deployments, 2025: scaling and multi-agent systems — vendor signals (Arthur, BrainTrust, Galileo, Arise), leaked mid‑April revenue chatter, and costly human validation rates seen in a Merkor spreadsheet ($50–$200/hr). Open source AnyAgent for multi‑agent testing exists.
AI-written summary · about
chapters (18)
- 0:00Introduction to Arthur AI and Mozilla AI
- 0:462025: The Year of Evals
- 1:15AI/ML monitoring and evaluation
- 2:48The Year of the Agent
- 3:26The need for 'evals' wasn't obvious to the C-suite
- 4:15Pre-ChatGPT launch
- 6:06Venture capitalists' predictions
- 7:03Macroeconomic side of things
- 8:06OpenAI launching ChatGPT
- 9:152023: The Year of GenAI
- 9:392024: GenAI applications in production
- 10:222025: Scaling and autonomy
- 11:35Definition of an agent
- 12:06Connecting to downstream business KPIs
- 14:40Shift to multi-agent systems monitoring
- 15:42Q&A
- 16:16Discussion on domain expertise in evaluations
- 18:13Discussion on LLMs as judges
related talks
01h 17mThe year of the AI Agent: Beyond Automation
01h 12mKeynote: The Age of Agents
52mThe Agentic Shift- Are We Reasoning, Or Just Retrieving Smarter?
40mYou, AI, and the Future of Society
19mRedefining Intelligence: How Reasoning Is Re-Shaping AI in 2025
18mNext-Generation AI Agents: Orchestrating Technological Evolution
14mReliable Agentic AI, the Open Way
01h 01mKeynote Panel: Future of AI agents