Agents reported thousands of bugs, how many were real?
summary
A new SM-100 benchmark uses 100 triaged, validated bugs from 84 public repos to test agentic maintenance, and a new agent called Bismuth leads the needle‑in‑a‑haystack results by finding 10 of those needles while the next system finds 7. Basic loops hit huge noise with a ~97% false positive rate, open‑source agents score 1–2% true positives, and some popular agents only score single‑digit percent on SM‑100; one issue generates 70 reported defects per run.
AI-written summary · about
chapters (9)
- 0:00Software agent bug detection intro
- 1:25Limitations of coding benchmarks
- 4:55SM benchmark for bug detection
- 8:49Benchmark methodology
- 10:33Basic agent performance
- 13:22Open-source vs commercial agents
- 15:52Narrow thinking impact
- 16:55Software maintenance implications
- 17:59Conclusion
related talks
44mTest Intelligence: Finding More Bugs in Less Time with AI
39mThe AI Arms Race in Vulnerability Management, Who’s Winning?
45m82 Bugs I Collected in a Year You Won’t Believe Made It to Production
28mOwn Goal: How AI Agents Are Quietly Scoring for the Other Side
44mSyzbot BoF: Open Discussion
42m82 Bugs I Collected in a Year You Won’t Believe Made It to Production
37mSyzbot: 7 years of continuous kernel fuzzing
40mCatching More Bugs with Fewer False Alarms