Coding Evals: From Code Snippets to Codebases
summary
Traces the shift from snippet benchmarks to whole-repo evaluation with concrete practices: dynamic LiveCodeBench releases (six versions) that use 30–50 inputs per problem to reduce contamination, automated fuzzed test generation, and sliding-time evaluation to detect training-set leaks. Repository optimization uses mined commits and 100+ optimization tasks with workload-based performance tests; a model-driven hack detector finds adversarial patches and reports a ~30% reward-hacking rate on early models. User studies (Copilot Arena, LMArena RepoChat) show latency >1s drops acceptance and require latency-controlled A/B comparisons.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:35Competition coding models
- 7:10High-performance generation
- 13:05Codebase translation
- 14:28Real-world evaluations
- 16:34Key takeaways and future directions
related talks
17mSWE-rebench: Lessons from Evaluating Coding Agents
28mSleep better on release day
25mEvaluating An LLM's Ability To Code
02h 44mMeasure, don’t guess: benchmarking stories from the trenches
49mEvals 101
48mRefactoring from Good to Great - A Live-Coding Odyssey
15mCI in the Era of AI: From Unit Tests to Stochastic Evals
50mPanel: Beyond The Vibe: Code Quality First