Benchmarking semantic code retrieval on Claude Code
summary
Benchmarks measure actual retrieval: a 50-task run on ContextBench compares raw Claude Code, windowed grep (reads capped at 50 lines), and windowed reads plus Turbopuffer-backed semantic search via Turbo Grep and the Voyage code model. Claude Code wastes ~1 in 3 file reads, windowed grep drops that to 1 in 5, and semantic search to 1 in 8, raising file precision from 65% to 87%. Semantic search wins on behavior-adjacent files; grep wins on import tracing. Cursor shows ~24% relative answer accuracy gain and 2.6% higher code retention.
AI-written summary · about
chapters (10)
- 0:00Semantic code retrieval intro
- 1:50Performance gains from search
- 3:55Cache compute savings
- 4:36Turbo Grep tool intro
- 5:19Benchmarking with ContextBench
- 7:02Precision results file relevance
- 8:04Recall analysis tool comparison
- 9:46Summary of findings
- 10:50Future of lightweight tools
- 11:12Q&A on semantic search
related talks
36mBeyond Grep: Search for Reliable Coding Agents
18mCoding Evals: From Code Snippets to Codebases
34mUnsession: "AI coding tools"
18mMastering Claude Code for Data Professionals
01h 06mHow Claude Code Works
11mCG-Bench: Can Language Models Assist Call Graph Construction in the Real World?
44mClaude Code for Spring Developers
14mHow we taught agents to use good retrieval