Fuzzing in the GenAI Era
summary
Claims a production-ready evaluation loop built from Haizing fuzzing plus agent judges, not static datasets: uses an agent library Verdict and RL (GPO) tuning to train 600M and 1.7B judges that reach ~80% on RewardBench, and a debate/self‑verify ensemble that outperforms larger models at lower cost and latency. Input search uses gradient-guided token flips, MCTS and embedding-space search. Case studies: uncovered jailbreaks in a major Hungarian bank loan app, hardened a Fortune 500 voice-agent pipeline, and raised human agreement by 38%.
AI-written summary · about
chapters (14)
- 0:00Introduction to Haizing
- 1:16The "Last Mile Problem" in AI
- 2:47The Brittleness of GenAI Applications
- 3:54Examples of Brittle Chatbots
- 4:29Inadequacy of Standard Evaluation Methods
- 6:09Haizing: Simulating the Last Mile
- 8:43Scaling Evaluation with Agents as Judges
- 9:29Verdict: Accuracy vs. Latency
- 11:47Scaling Evaluation with RL-Tuned Judges
- 14:06Fuzzing vs. Adversarial Testing in AI
- 14:37Simulation as Prompt Optimization
- 16:23Case Study: Haizing a Major European Bank's AI App
- 17:05Case Study: Haizing a F500 Bank's Voice Agents
- 17:46Case Study: Scaling Voice Agent Evals with Verdict
related talks
43mEval-Driven Development: Mastering Agentic Tracing and Expert-Aligned Judges With MLf...
46mEnhancing Agent Reliability Through Evaluation-Driven Development
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
22mEnabling Security Defenders with AI- 02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
44mTL;DR: Applying AI to Security
19mKeynote: Towards Building Safe and Secure Agentic AI
46mAI Testing Beyond the Basics: Ensuring Truthful and Reliable Agents