AI Applications Security Puzzle
summary
Live product failures: assistants obeying injected commands, a vending machine convinced to sell at negative margins, and fine‑tuned APIs hit with thousands of queries that enable model extraction. Cites OWASP AI Testing Guide and Google SAIF, a 44% YoY rise in public‑facing exploits, and the LiteLLM GitHub‑credential compromise. Practical defenses: input validation, separate instruction channel, output validation, rate limits and watermarking, SBOMs and credential hygiene, and treating every RAG document as potentially malicious so tests catch these exact failures.
AI-written summary · about
chapters (60)
- 0:00Start
- 0:05Session Opens - The AI Applications Security Puzzle
- 0:35The Setup: An Assistant Told to Ignore Its Instructions - and It Does
- 1:05These Are Live Products, and Nobody Wrote a Test for It
- 1:35Speaker Intro - Maryia Tuleika, Quality Engineering Lead, Regent AB
- 2:37Audience Poll: Who Has Actually Security-Tested the AI Tools They Use?
- 5:14How a Quality Engineer Ended Up in Security
- 6:16The Hiking Habit: Scan the Whole Terrain, Not Just the Next Step
- 7:16Seeing Systems as Trust Boundaries Instead of Things to Test
- 7:47Starting With API Security - and Then LLMs Arrived
- 8:49The Same Patterns Reappearing, Faster and at Greater Scale
- 10:21Meet Anna - The Junior Engineer Who Ships Fast
- 11:52People Who Build Fast Aren't the Problem; They Need the Right Tools
- 12:56Why It's a Puzzle: Missing Pieces Give You False Confidence
- 13:26Four Layers, Each With Its Own Vulnerabilities
- 13:58The Two Frameworks: Google SAIF and the OWASP AI Testing Guide
- 14:58The Map: Application, Model, Infrastructure, Data
- 16:34Stop One - The Application Layer, the Customer-Facing Surface
- 17:04Prompt Injection: Number One on the OWASP Top 10
- 18:07Getting an AI to Follow Your Instructions Instead of Its Own
- 19:41Why Blending System Prompt and User Input in One Context Window Breaks It
- 20:11Direct vs Indirect Prompt Injection
- 20:43Instructions Hidden in Documents, Emails and CVs
- 21:15The Vending Machine That Was Talked Into Negative Margins
- 22:48Not One Message - A Carefully Constructed Chain of Conversations
- 23:19Each Message Moving the Boundary a Little Further
- 24:56Defence One: Input Validation - The Same Thinking as SQL and Web
- 25:30Defence Two: Separate the Instruction Channel From the Data
- 26:00Defence Three: Output Validation and Deterministic Guardrails
- 26:32Defence Four: Least Privilege - Don't Hand Over the Master Keys
- 27:06Stop Two - The Model Layer, and Why Brains Are Attractive Targets
- 28:07Anna Publishes a Fine-Tuned Model and the API Calls Pour In
- 28:41Model Extraction, Stealing and Distillation - One Idea, Three Names
- 29:46Thousands to Millions of Queries, Then Build the Replica
- 30:17Why It Looks Exactly Like Normal Traffic - No Errors, No Alerts
- 31:23Documented Cases: AI Companies Suing Each Other Over Stolen Outputs
- 32:26Watermarking as the Vendor Response
- 33:31What Anna Can Actually Do: Rate Limiting Per API Key
- 34:02Anomaly Detection and Systematic Boundary Probing
- 34:32Output Perturbation: Calibrated Noise to Disrupt Extraction
- 35:02Stop Three - The Infrastructure Layer, the Easiest to Overlook
- 36:05Supply Chain Attacks Target What You Trust, Not You
- 36:38Why AI Tooling Is Especially Exposed: New, Fast-Moving, Under-Resourced
- 37:11SAIF Calls Out Supply Chain Integrity as a Critical Requirement
- 37:41The LiteLLM Incident: Compromised GitHub Credentials
- 38:15A Widely Used Proxy Library and a Very Large Blast Radius
- 39:18Defence: Dependency Management and the SBOM as a Standard
- 40:20Credential Hygiene and the Vendor Assessment Everyone Skips
- 40:51Stop Four - The Data Layer, Where Attacks Are Most Invisible
- 41:23Data Poisoning Without Any Visible Intrusion
- 41:54Poisoning Training Data, RAG Documents and Tags
- 42:27Hidden Instructions in Internal Documents the Assistant Reads
- 42:57Defence: Treat Every Document Feeding Your AI as Potentially Malicious
- 44:01The Complete Puzzle - Every Case Real, and Every Case Addressable
- 44:32Write Down One Thing You'll Do Differently Today
- 45:05You Don't Need Security in Your Job Title, Just Curiosity
- 45:37Resources: The OWASP AI Testing Guide and Google SAIF
- 47:12Q&A: How Autonomous Agent Risk Differs From Plain LLM Risk
- 50:25Q&A: How QA and Security Teams Should Collaborate on Threat Modelling
- 52:30Q&A: Risks When Agents Execute Tools Autonomously
related talks
55mAI Security in Practice: Protecting Your AI-Powered Applications
54mA Practical Guide to AI Threats & Mitigation Strategies
37mFundamentals of AI safety and security
44mAI Showdown: The Good, The Glitchy, and The Risky
31mSecurity Considerations for Services Using AI Models
45mSecuring AI: There Is No Try, Only Do!
53mHow AI is impacting cybersecurity. Does it?
59mPentesting AI: How to Hunt a Robot