How to Break AI Systems (Before Someone Else Does)
summary
Claims that production AI routinely fails where standard security testing misses prompt injection and agent goal manipulation, and shows live demos of hidden prompts in documents, agents exfiltrating data via rendered images and markdown, and privacy leaks. Presents open source tooling: Pirate (includes Crescendo templates), mentions Nvidia Garrick, and gives a hands‑on platform at arena.tinycode.io with code ndci2025 and about 50 challenges. Recommends spotlighting mitigations (limit, data‑mark, encoding like B64) and notes B64 preserves GPT‑4 accuracy while degrading GPT‑3.5.
AI-written summary · about
chapters (23)
- 0:00AI security and red teaming
- 1:06AI security risks critical
- 2:10AI vs traditional red teaming
- 3:57AI for security vs security of AI
- 5:19Responsible AI harms
- 8:55AI system failures examples
- 11:04AI system architectural flaws
- 13:50AI attacks easier than exploits
- 15:15Prompt injection attacks
- 17:23Jailbreaks via competing objectives
- 19:49Jailbreak templates effectiveness
- 21:13Crescendo multi-turn attack
- 23:19Indirect prompt injection RAG poisoning
- 25:06Attacking AI agents
- 27:36Computer usage agents risks
- 29:23AI red teaming tools
- 31:10Pirate tool for red teaming
- 34:25Converters and scoring in red teaming
- 39:41Mitigations for prompt injection
- 41:07Spotlighting to prevent jailbreaks
- 45:24Technical controls for AI security
- 46:26Hands-on AI hacking walkthrough
- 47:31Q&A
related talks
37mHow to Break AI Systems (Before Someone Else Does)
19mKeynote: Towards Building Safe and Secure Agentic AI
36mAgentic AI Under Attack: Live Demos of Exploits Through Autonomy and Trust
01h 01mHow to Break AI Systems (Before Someone Else Does)
37mKeynote - Hacking AI How to Survive the AI Uprising
45mSecuring AI: There Is No Try, Only Do!
18mDefending the unknown using AI and cybersecurity
01h 03mBuilding Secure Infrastructure for Productive AI Agents