Evaluating An LLM's Ability To Code
summary
Describes a practical eval suite called go toe eval that moves beyond single-file benchmarks to multi-file, real-world tasks (examples: 13 commits, 14 files changed, ~2,200 lines added) and measures tool+model stacks across languages including C++, C#, Kotlin, Java, Rust and Unity 3D. Uses static code analysis plus unit tests and an LLM-as-judge broken into yes/no checks to reduce scoring variance to about 5%. Runs ~20 active evals, five models each, five iterations to smooth nondeterminism, and highlights toolchain limits (Cursor, Kira, OpenRouter) and provider/temperature fragility.
AI-written summary · about
chapters (6)
- 0:00Introduction and background
- 5:02Why existing benchmarks fall short
- 10:38Version one of GoToEval
- 16:35Version two improvements
- 20:27Future of LLM evaluation
- 24:22Closing and resources
related talks
33mPractical Evaluation of LLMs for Code
38mMaking LLMs reliable - A practical framework
35mEvaluating LLM systems: the nuts and bolts
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
32mA Practical Guide to LLM Evaluation
44mPractical LLM Evals: Down-to-Earth Tooling for Everyone
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications