SecureMind: A Framework for Benchmarking Large Language Models in Memory Bug Detection(…)
summary
A Python framework that automates dataset retrieval, cutoff-date filtering to avoid data leakage, probe orchestration and metric calculation, SecureMind evaluates 10 LLMs (including 7 reasoning models) across 16K samples covering 8 CWE memory types. It ships 24 probes (14 detection, 10 repair), three metric classes (deterministic, performance, reasoning via cosine similarity), and shows chain-of-thought boosts ≈10% and CWE-context boosts ≈5–10% for detection, but assembly reading scores ≈0 and repair success ranges 3–37%, so detection is moderate and repair remains weak.
AI-written summary · about
chapters (8)
- 0:00SecureMind introduction
- 1:30LLM vulnerability detection background
- 4:44SecureMind design overview
- 8:37Evaluation setup with SecureMind
- 10:03Memory bug evaluation insights
- 14:19Bug repair challenges with LLMs
- 15:26Summary and benefits
- 16:09Q&A
related talks
23mSecurity and auditing tools in Large Language Models (LLM)
38mMaking LLMs reliable - A practical framework
46mHow Large Language Models Are Reshaping the Cybersecurity Landscape
44mI Abuse Your LLM Live with 10 Security Vulnerabilities
31mEnhancing Secret Detection in Cybersecurity...
15mInterleaving Large Language Models for Compiler Testing
22mEnabling Security Defenders with AI
11mCG-Bench: Can Language Models Assist Call Graph Construction in the Real World?