Never Trust a Monkey: The Chasm, the Craft, and the Chain of AI-Assisted Code
summary
Claims a concrete pipeline to make AI code trustworthy: Intent Integrity Chain that runs prompt → spec → test → code, with specs human-reviewed, tests generated deterministically from specs, and assertions hash asserted and stored to prevent tampering. Cites studies showing ~50% of LLM code is incorrect and only a few percent fully trust models, plus anecdotes of 25,000-line outputs with passing tests but broken apps. Uses tools like Specit and an intent integrity kit and emphasizes never letting the same model both produce and verify results.
AI-written summary · about
chapters (21)
- 0:00AI-assisted code trust issues
- 2:40Intent-to-code chasm
- 7:41Why design docs fail
- 12:17Trust problem in dev
- 15:27AI models' limits
- 17:56The 'looks good to me' syndrome in code reviews
- 19:02Spec-driven flaws
- 22:37Critique of spec-driven dev
- 24:47Tests guard AI code
- 27:43Circular verification in TDD
- 30:14Why BDD failed
- 32:23Monkeys vs algorithms
- 33:24Locking assertions
- 36:13Intent integrity chain
- 38:00Handling unknown APIs
- 40:29Quality context for AI
- 42:17Intent Integrity Kit
- 44:27Q&A: Test derivation
- 45:51Q&A: Intent Integrity vs waterfall
- 48:50Q&A: Test quality focus
- 50:38Closing remarks
related talks
48mNever Trust a Monkey: The Chasm, the Craft & the Chain of AI-Assisted Code
54mNever Trust a Monkey: from AI Gobbledygook to Deployable Code
32mCan We Trust AI-Generated Code?
26mThe Trust Problem: Designing Quality Frameworks for AI-Generated Code
39mBack to Basics: Cutting Through the AI-Generated Code Noise
34mTrust in the Age of AI Agents | Building Trustworthy AI
24mHow to Build Trustworthy AI
33mShipping Fast, Breaking Silently: Why AI-Generated Code Needs a Second Opinion