Interleaving Static Analysis and LLM Prompting
summary
A static analyzer (EESI) calls an LLM whenever a function’s error-specification lattice value is bottom, and the LLM output is converted to EESI’s abstract domain and re-fed into analysis; EESI interleaved with GPT‑4 Turbo raises average recall from 52.55% to 77.83% and F1 from 0.612 to 0.804 while keeping precision near 86% (86.67% → 85.12%) across benchmarks (MbedTLS, zlib, pigeon, netdata, httpd, Apache). Prompts include abstract domain, few‑shot examples and Chain‑of‑Thought, and answers are validated by success‑value and return‑range checks; GPT‑4 adds ~30 min vs ~7 min static only.
AI-written summary · about
chapters (13)
- 0:00Interleaving static analysis with LLMs
- 1:01Static analysis and LLM prompting
- 3:29Interleaving static analysis with LLMs
- 4:34Error specification with static analysis
- 6:37Designing prompts for error specs
- 8:03Handling LLM hallucinations
- 9:08Case study: embed TLS and third-party functions
- 10:54Limitations of LLMs alone
- 12:21Research questions and metrics
- 13:34Experimental setup and benchmarks
- 14:39Results vs baseline static analysis
- 16:25Conclusion and findings
- 17:31Q&A
related talks
18mEnhancing Static Analysis for Practical Bug Detection: An LLM-Integrated Approach
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
25mEvaluating An LLM's Ability To Code
18mBeyond Static Pattern Matching? Rethinking Automatic Cryptographic API Misuse Detection(…)
13mHallucination-Resilient LLM-Driven Sound and Tunable Static Analysis
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
47mHarnessing LLMs for Next-Level Security Analysis