Rectifying Privacy and Efficacy Measurements in Machine Unlearning...
summary
RULI introduces a dual-objective attack that measures both unlearning efficacy (does the unlearned model match a retrain reference) and privacy leakage (does it match a never-trained held-out distribution) at per-sample granularity. The method trains dozens of shadow models and a simple calibration test router to correct output suppression, and it finds stark gaps: on vision benchmarks RULI reaches up to 69% membership-inference success distinguishing unlearned from retrained models, while sparse fine-tuning can show low TPR (≈10% at 1% FPR) yet still produce high attack success, exposing underestimated vulnerabilities.
AI-written summary · about
chapters (14)
- 0:00Machine unlearning evaluation intro
- 1:15Current unlearning eval limits
- 3:21Privacy regulations framework
- 4:27Privacy and efficacy requirements
- 6:13Privacy leakage challenges
- 8:00Efficacy and output suppression
- 9:27Per-sample evaluation method
- 11:40Roly evaluation framework
- 13:52Vision datasets results
- 14:56Vulnerable samples significance
- 15:38Membership inference attacks comparison
- 16:01Text datasets and generative results
- 17:05Privacy vs unlearning auditing
- 17:46Conclusion and future
related talks
30mUnforgettable, that's what you are: Evaluating Machine Unlearning and Forgetting
15mLearning and Unlearning Your Data in Federated Settings
23mHow can a learnt ML model unlearn something: Framework for "Machine Unlearning"
28mPrivacy preserving machine learning
01h 08mUnwanted Learning: What It Is, Why It Matters, and What To Do About It
54mLearning Uninformative Representations
17mData privacy and reliable learning
01h 17mTutorial: Privacy, Part I