AI in Testing: Hip or Hype?
summary
Begins by explaining how LLMs work conceptually, then tests specific claims: debunks a 10x productivity claim, shows real gains around 3–6% productivity for DOM-finding and repetitive automation, demonstrates LLMs generating large test datasets (successfully produces 10,000 names, struggles with cascading transforms), compares scripted tools (Splunk) to fuzzy log-file matching, and evaluates LLMs for defect prediction, test-case generation, and code scaffolding with practical caveats.
AI-written summary · about
chapters (15)
- 0:00AI in software testing intro
- 1:49Digital agents productivity claims
- 6:54How LLMs work
- 10:27Gartner hype cycle for AI
- 15:58AI’s role in testing
- 20:35Fairness in AI testing
- 22:22Summarization and synthesis with LLMs
- 25:09AI log file analysis
- 30:36Defect prediction with AI
- 33:06LLM test data generation
- 38:29LLMs writing test code
- 41:47Novel test ideas with LLMs
- 46:21LLM test case generation
- 48:50Automating tasks with AI
- 50:56AI testing takeaways
related talks
33mAI is coming for your job
32m🧐 A Look at Possibilities of Using AI in #SoftwareTesting
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
25mAI vs. AI: How to Test AI Systems Using AI
50mHow to Test LLM Agents
35mEvaluating LLM systems: the nuts and bolts
01h 04mImpact of AI on #SoftwareTesting: Are Testers Ready? 🔮
22mFrom the Trenches: Automated Testing of Large Language Models