Enabling Security Defenders with AI
summary
Benchmarks measure concrete capabilities: Autopatch Bench asks an LLM agent to read OSS‑Fuzz stack traces, produce a patch that compiles in Docker, and survive fuzzing while preserving functionality; verification uses compile checks, fuzz regression, and manual/judge‑LLM review. A separate cybertock eval suite scores SOC tasks like incident triage, malware sandbox reasoning, and threat‑report analysis. Real results include weak baseline LLM exploit performance until a Google agent with debugger/tools maxes toy challenges, and heavy QC on Arvo and synthetic data.
AI-written summary · about
chapters (9)
- 0:00AI security at Meta
- 1:26AI risks in cyber security
- 3:56Mitigating risks with guardrails
- 5:59AI as adversary tool
- 7:22Measuring risks with benchmarks
- 11:20Introducing Autopatch Bench
- 17:04Cybertock eval for security ops
- 18:30Challenges in AI security benchmarks
- 21:21Closing and hiring
related talks
44mTL;DR: Applying AI to Security
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
34mLLM Security 101 - An Introduction to AI Red Teaming
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
01h 02mPanel Discussion:The Present and Future of AI and Security
05mLightning Talk: AI Cyber Challenge
01h 01mIntroduction to AI Security
20mSecurity Threats in Modern LLM Applications