Confidently Wrong: When AI Cannot Catch Its Own Bugs
Talkat Open Source Summit India 2026 · on Wednesday, 2026-06-17· at 16:25· Jasmine Hall 2 (Level 3)·
summary
Models regularly produce simple logic bugs—arithmetic, timezones, unit conversions—and then verify their own wrong answers. An empirical study using Queen 30B generator and cross-checkers Granite 30B, Llama 70B shows self-verification detects ~0% of silent failures, cross-model checks catch 17–30%, while deterministic oracles (calculator, Python zoneinfo, compiler, SAT solver) catch >95%. Agent trajectories from SWEAGENT launders errors into later stages (≈95% used as patches), with agents claiming 87% success but external checks showing 33%. Practical fixes: execution-trace grounding, Marya.ai style validators, and independent deterministic checks.
AI-written summary · about
chapters (9)
- 0:00AI verification challenges
- 2:29Correlated judgment in AI
- 7:25Silent failures in verification
- 9:33Trap scenario in reasoning
- 15:19Why verifiers fail
- 23:22Error laundering in AI agents
- 32:08Breaking verification circularity
- 36:39Independent verification design
- 40:31Key takeaways
related talks
23mBeyond Tests: What to Verify in AI-Generated Code
32mCan We Trust AI-Generated Code?
20mThe Three Ways AI Agents Fail. Here's How to Fix Yours.
23mBeyond Tests What to Verify in AI Generated Code
28mAll Your AI are Belong to Us
01h 04mWhat are we thinking in the age of AI?
29mTesting for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems
32mEscape the Try-Again Loop: Verifying AI-Generated Code