AI powered entomology: Lessons from millions of AI code reviews
summary
Analysis of millions of automated code reviews from Graphite’s Diamond shows LLMs can reliably flag certain bug classes but also flood PRs with unwelcome best-practice noise; experiments with Claude and other models plus prompt tuning cut hallucinations using upvote/downvote feedback and a <4% downvote rate, and measuring whether comments produce edits finds an improving 52% change rate (March). Concrete tooling: prompt filters that only emit checks humans accept, realtime emoji signals, and automated tracking of PR edits.
AI-written summary · about
chapters (6)
- 0:00Introduction to talk
- 0:42AI code reviews and bugs
- 2:06LLM limitations and frustrations
- 3:50Classifying AI review comments
- 6:38Measuring AI review success
- 9:02Human vs AI comment effectiveness
related talks
26mAI-powered Code Review
19mThe AI Code Review Bottleneck: Escaping the Velocity Trap
50mIdentifying and fixing Issues in Code using AI based tools
01h 01m"Looks Good to Me": A Practical Guide to Handling AI-Generated Code
23mAI assisted code reviews Quality and Security at Scale
09mImproving code reviews with AI agents
34mAI and Code Quality: Building a Synergy with Human Intelligence
07mLightning Talk: Shift Left With AI: Smarter Code Reviews for DevOps...