What are we thinking in the age of AI?
summary
Shows empirical failures with LLMs: 480 attempts to generate simple JSON test data produce a 57% failure rate, 60 iterations reveal frequent self-repudiation, and repeated prompts yield inconsistent SQL types and garbage diagrams. Notes unreliable outputs from ChatGPT, GPT‑4o, Bing Chat and copilots, flaky test tools like Mabel and Katalon, and many demo-only results versus real experiments. Argues for algorithmic obscurity, radical fragility, and higher scepticism plus human-led experiments and checklists before deployment.
AI-written summary · about
chapters (8)
- 0:00Keynote on AI and testing
- 1:24Testers in the age of AI
- 11:49Clever Hans and AI deception
- 23:23Algorithmic obscurity challenges
- 35:17Developers' testing limitations
- 41:18Mistrusting AI test tools
- 55:32AI failures in test artifacts
- 1:01:00Call to action: Don’t outsource smarts
related talks
50mPanel Discussion: What Can Go Wrong With AI in Testing?
32m🧐 A Look at Possibilities of Using AI in #SoftwareTesting
13mNew Challenges in Software Quality Brought by GenAI and LLMs
19mTesting LLM-Powered Applications
37mGenerative AI: Beyond (Just) Hype
46mEnhancing Agent Reliability Through Evaluation-Driven Development
48mMastering the AI Shift: Adaptive Strategies for Software Evolution
01h 28mFuture of Software Programming with AI Bots at Hand