Generative AI: Say Goodbye to Deterministic Testing?
summary
Generative AI introduces non-determinism so deterministic unit tests fail; practical responses include external tool verification (parsers, compilers, security scanners) and using property-based tests like Hypothesis to define invariants, plus unit benchmarks that measure specific behaviors statistically. Small local models (Llama, IBM Granite) produce usable code often about 80% of the way there; large models act as a LLM-as-judge in testing or to synthesize synthetic datasets. Final layer uses statistical confidence (think sigma thresholds) to decide pass/fail.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:49Generative AI breaks testing
- 6:48Test-driven development and determinism in software
- 8:55Functional programming & property tests
- 11:22Non-deterministic AI outputs
- 13:51Adapting to AI’s non-determinism
- 14:13Coupling in AI apps
- 15:59Model strategy & prompt engineering
- 20:09Acceptance testing with models
- 20:31Verifying AI with tools
- 25:51Adapting benchmarks for unit tests
- 26:56Benchmarks for AI testing
- 28:22Translating requests to API calls
- 29:46LLMs as judges
- 31:52Statistical analysis for AI
- 34:18Rethinking testing with AI
- 36:48Tuning models instead of code
- 38:13Q&A
related talks
09mTesting Strategies for Generative AI Applications
46mTDD & generative AI - a perfect pairing?
45mTDD & generative AI - a perfect pairing?
01h 02mBreaking the Black Box: Why Testing Generative AI Is Full Spectrum
41mAI-Augmented Testing: How Generative AI, Prompt Eng Turn Testers Into Superheroes
51mTesting Practices for Generative AI Systems: Focusing on Test Design and Analysis
29mImpact of #GenerativeAI in Development and #SoftwareTesting
47mTDD & Generative AI: A Perfect Pairing