Prompt Hardening - Secure Code Generation Using AI
summary
A reproducible evaluation harness uses PromptFoo and OpenRouter-connected models (Gemini 1.5 Pro, Claude Opus, GPT‑3.5 Codex, GPT‑4o mini, GPT‑5.1/5 mini) and an LLM-as-a-judge prompt to scan 3,000 generated code samples from 300 unique prompts; prefixing prompts with using best practices raises pass rates from ~68% to ~83% (≈15.3% absolute improvement). Tests uncover concrete failures: hardcoded credentials, missing rate limiting, and no least‑privilege, and the evaluation artifacts and samples are stored for automated CI checks.
AI-written summary · about
chapters (15)
- 0:00Prompt hardening introduction
- 3:17AI secure code research journey
- 7:09How LLMs learn and security limits
- 9:17Context drift and prompt limits
- 11:25Shorter prompts for better security
- 16:28Prompt prefix method
- 19:34Evaluations with Promptfoo setup
- 25:58First experiment results
- 27:44Second experiment with realistic prompts
- 30:19Demo of evaluation results
- 33:22Model security testing surprises
- 35:28Failed security evaluations analysis
- 41:11Future of secure AI code
- 43:19Practical prompt tips
- 45:05Q&A
related talks
58mPrompt Hardening - Secure Code Generation Using AI
02h 01mPrompt Engineering and AI Red Teaming
44mThe Science of Effective Prompting
29mSurvival of the Safest: Automating LLM Defense with Genetic Programming
59mUsing AI for Secure Code Creation: Enhancing Software Security
01h 08mIn Prompts We Trust
29mOne-Click Code Fix: Securing Code Using AI
16mPrompt Engineering for Testers: Types and Applications