Automating Technical Privacy Reviews Using LLMs
summary
Proof-of-concept assesses 80 HERE and 60 Uber architecture documents (half labelled containing at least one privacy risk) using GPT-4 and locally deployed Mistral Mixr and Llama 3. Pipeline extracts technical summaries, measures factuality and F1 (precision/recall) against human labels, finds models reduce triage workload but produce false positives and occasional factual hallucinations; gated prompts (decision-tree style) push performance up to 90% accuracy at much higher computational cost, so human review remains required.
AI-written summary · about
chapters (7)
- 0:00Privacy review automation intro
- 2:59Challenges in privacy reviews
- 4:02Experimental setup
- 7:12Single vs multi-step LLM
- 8:59Evaluating model accuracy
- 10:47Cost, scalability, future
- 13:42Conclusion
related talks
29mAutomating Privacy Reviews with LLMs: Challenges & Opportunities
16mPrivacy Review for Non-Maniacs
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
42mHuman in the loop is not enough
39mThreat Modeling Using LLMs
28mTrustworthy LLMs: Vibe checks are not all you need
38mMaking LLMs reliable - A practical framework
15mTurning Privacy Risk Assessment Into 20 Questions for Developers