Beyond Static Pattern Matching? Rethinking Automatic Cryptographic API Misuse Detection(…)
summary
Automatic LLM detection of cryptographic API misuse shows serious instability: over half initial reports are false positives, yet applying a repeated-response validation technique boosts recall to nearly 90% and cuts false alarms when the detection scope is focused. Evaluation covers commercial and open-source LLMs (including GPT-4 variants) against handcrafted and Apache Java benchmarks, finds 63 new vulnerabilities (47 confirmed, 7 already fixed), and documents frequent failures from wrong crypto knowledge and code-semantics misunderstandings.
AI-written summary · about
chapters (11)
- 0:00Cryptographic API misuse intro
- 1:51Pattern-based detection limits
- 4:02LLM-based detection opps & challenges
- 5:26LLM misuse detection eval
- 7:55LLM misuse detection effectiveness
- 9:02False positives in LLM detection
- 11:10LLM misuse error categories
- 12:35LLM vs static detectors
- 14:25LLM failure patterns
- 15:51Multilingual LLM validation
- 17:17Conclusion & future work
related talks
36mHow to statically detect insecure uses of crypto...
20mSecurity Threats in Modern LLM Applications
01h 01mIntroduction to AI Security
29mAutomating Privacy Reviews with LLMs: Challenges & Opportunities
47mHarnessing LLMs for Next-Level Security Analysis
47mUsing the CGC’s fully automated vulnerability detection tools
36mUsing AI to Discover Silently Patched Vulnerabilities in Open...
46mGuarding the LLM Galaxy: Security, Privacy, and Guardrails in the...