Spec-Driven Testing for Agents With A Brain the Size of A Planet
summary
Claim: a malicious instruction in a poem jailbreaks large models but not small ones because bigger models parse and execute the hidden instruction. Practice: use spec driven validation — explicit rules like never offer more than 10% discount, domain ontologies, rights and roles, and robustness requirements (typos, rephrasings) so tests survive model swaps. Outcome: a platform is built to generate edge cases, run security checks and robustness sweeps against deployed agents.
AI-written summary · about
chapters (8)
- 0:00Safe intelligence and ML validation
- 1:30Spec-driven testing importance
- 2:33Large AI model challenges
- 4:41Agent specification components
- 7:05Robustness and real-world testing
- 8:48Fair evaluations and integration tests
- 10:56Automated agent testing loop
- 11:58Future directions for agent specs
related talks
50mSpec-driven Development using Coding Agents
23mBeyond Tests: What to Verify in AI-Generated Code
47mSpec-Driven Development with AI Agents From High-Level Requirements to Working SW
38mCoding Agents & Language Evolution: Navigating Uncharted Waters
23mBeyond Tests What to Verify in AI Generated Code
32mSpec Driven Dev From Single Player to Multiplayer to Ecosystem
01h 04mSpec-Driven Development: Agentic Coding at FAANG Scale and Quality
48mSecuring Agentic AI