Malleable Evals: Why Are We Evaluating Adaptive Systems with Static Tests?
summary
Claim: static benchmarks fail because agents adapt under test. Evidence: an 80% stable 20% breaks rule, examples of harnesses that self-curate test suites from runtime traces, and telemetry-driven harnesses in Open Claw plus features in Claude and Codex. Tools actually used include auto optimization scripts (Kapathy's auto research) and Python tuning loops. Built outcomes are always‑on eval services and telemetry in the loop that detect regressions and self-correct; some prototypes run but full production maturity is ongoing.
AI-written summary · about
chapters (7)
- 0:00The state of AI evaluation
- 1:56Limitations of static testing
- 4:22Moving toward adaptive evals
- 5:36Evolution of AI engineering
- 7:22Intent engineering concepts
- 10:01Implementing dynamic evals
- 12:13Future of malleable evals
related talks
38mBuild Evals That Actually Matter
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
47mBuild Trustworthy AI Agents powered by Evals
46mEnhancing Agent Reliability Through Evaluation-Driven Development
18mCoding Evals: From Code Snippets to Codebases
06mWhy should anyone care about Evals?
09mHow to run Evals at Scale: Thinking beyond Accuracy or Similarity
37mAI agents testing How to evaluate the unpredictable