Breaking the Black Box: Why Testing Generative AI Is Full Spectrum
summary
Black‑and‑white pen testing fails for generative AI because models behave probabilistically, hallucinate, and require behavioral persuasion rather than code exploits; validation requires access to backend data so findings aren't just convincing fabrications. Practical techniques include context poisoning, crescendo and crop attacks, adversarial suffixes, and permission/goal manipulation against agents that can take real actions. Demonstrated tooling and frameworks include Crop Duster, Tapper, and open tools like Garac; recommended approach is AI‑driven, continuous red‑teaming integrated into CI/CD with custom, semantic tests rather than naive string filters.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 2:37Traditional vs AI testing
- 4:46AI testing lacks clear results
- 7:14Stochastic nature of AI
- 9:41AI security testing context
- 11:27Prompt injection and leakage
- 14:39What to test in generative AI
- 16:25AI red teaming as social engineering
- 19:40Common AI attack techniques
- 22:30Direct vs indirect prompt injection
- 25:43Context poisoning and contamination
- 27:49Overriding system instructions
- 29:16Advanced attack methods
- 34:54Testing agentic systems
- 37:21AI red teaming tools
- 41:14Live demos of attack tools
- 51:50Evaluating open-source frameworks
- 55:04Shifting left with AI controls
- 57:32Managing AI risks
- 58:38Future of continuous red teaming
- 1:01:00Q&A
related talks
09mTesting Strategies for Generative AI Applications
46mGenerative AI: Say Goodbye to Deterministic Testing?
55mAI Showdown: The Good, The Glitchy, and The Risky
31mTesting, Evaluating and Debugging Generative AI and Agentic Ap...
50mOh, the Possibilities: Balancing Innovation and Risk with Generative AI
46mGenerative AI: Cyber resiliency and risk
54mEnhance AppSec: Generative AI integration in AWS testing
44mGenerative AI Security — A Practical Guide to Securing Your AI Application