Unit testing LLM Agents
summary
Proposes a practical test setup that runs the agent in a scripted conversational scenario by pairing an actor LM (simulates the user, given a task and stop conditions) with an evaluator LM that scores the whole conversation and returns JSON {success, score, reason}. Implements a Python loop that exchanges messages until the actor signals stop, then asks the evaluator to validate; runs the same scenario many times with temperature sampling (example: 100 runs) to improve coverage. Works in hobby projects, JSON can be flaky so use schema enforcement and a stronger model for evaluation.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:10What are LM agents
- 3:00What is unit testing
- 4:45Problem with unit testing LLMs
- 7:15Using LLM as evaluator
- 9:03Testing LM agents
- 13:08Actor LM prompt design
- 15:35Evaluator LM prompt design
- 16:22Leveraging temperature
- 17:05Conclusion
- 17:45Q&A session
