Practical Evaluation of LLMs for Code
summary
Open-weight code LLMs are tested inside IDEs (JetBrains, VS Code) with a plugin that records 2,700 downloads and 500 users, yielding about 600,000 data points across 12 languages; acceptance rates are low, Encoder (bidirectional) outperforms uni-directional models, and 66% of failures are model errors while 24% are application/context issues. Benchmarks review finds 273 datasets, shows saturation and contamination, and releases the Hamurabi code dataset and tools to refine benchmarks.
AI-written summary · about
chapters (9)
- 0:00Evaluating LLMs for code
- 2:28Online evaluation in IDEs
- 9:11Offline evaluation use cases
- 11:22Benchmarking challenges and gaps
- 15:52Finding and refining benchmarks
- 18:44Data contamination and HIPP dataset
- 20:28Harmfulness in code generation
- 25:20Long code evaluation progress
- 26:45Q&A
related talks
25mEvaluating An LLM's Ability To Code
16mThe LLM Leaderboard: Benchmarking AI Coding Models
10mWhat goes into the making of LLMs
35mEvaluating LLM systems: the nuts and bolts
38mMaking LLMs reliable - A practical framework
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
47mValidating LLMs – A True Story
40mLLM for Coding, the State and lnitiative