How We Red-Teamed Our Own AI Agent: Lessons from... (Josiah Peedikayil, HS)
summary
Operation Pale Fire red-teams Goose, Block’s open-source AI agent and demonstrates an end-to-end chain: sending external invites via Google Calendar MCP, hiding a malicious payload with zero-width Unicode smuggling inside a calendar description, and triggering Goose’s developer shell tool to run a curl | bash info‑stealer. Initial calendar campaign fails due to model variance, short context windows and a 50-invite/day rate limit; pivot to recipe-based system-prompt injection and a social-engineered developer run succeeds briefly before detections catch the exfiltration. Mitigations include stripping nonstandard unicode, surfacing decoded recipes, command allowlisting and prompt-injection detectors.
AI-written summary · about
chapters (12)
- 0:00Introduction and team acknowledgments
- 1:04What is a red team operation
- 2:29Introducing Goose AI agent
- 4:14Why target Goose for red teaming
- 6:01Initial access via Google Calendar MCP
- 9:28Prompt injection challenges
- 11:35Campaign one execution and failure
- 16:26Campaign two using recipes
- 19:55Google Meet social engineering
- 21:42Final payload and infostealer
- 24:47Mitigations and fixes
- 27:15Q&A session begins
related talks
27mThe accidental AI revolution
24mGoose: an open source AI agent
33mAgentic ProbLLMs: Exploiting AI Computer-Use and Coding Agents
27mKeynote: Towards Building Safe and Secure Agentic AI
20mAI Red Teaming and Automation: Exploring Societal Risks in...
12mCodename Goose: An open source local AI agent built on MCP
19mKeynote: Towards Building Safe and Secure Agentic AI
01h 01mAI Agents In-Depth – Function Calling, MCP and Tool Use Under the Hood