Large-Scale Private Learning, Part II
summary
Noisy SGD uses per-example gradient clipping and Gaussian noise with privacy amplification by sampling to get utility near nonprivate models; theoretical excess risk scales roughly like sqrt(d/(ε n)) up to polylog factors and tight bounds need ~n^2 iterations for the worst analysis. Privacy accounting uses the moments accountant to avoid paying log(1/δ) twice. An alternative is PATE (500 teacher models → aggregated private labels; ~500× computational cost) and practical results include a Reddit RNN with 40,000 vocab and ~18% next-word accuracy at ε≈2.
AI-written summary · about
chapters (10)
- 0:00Intro
- 2:25Stochastic Gradient Descent SGD
- 7:32Privacy Analysis
- 10:21Proof
- 19:12modern machine learning
- 25:46noisy SGD
- 38:46Markov Zone
- 42:36Private Segregation
- 46:06Privacy Cost
- 49:03Language Model
related talks
31mIs your model private?
01h 14mTutorial: Privacy, Part II
53mPrivate Learning with Auxiliary Public Data
01h 14mTutorial: Privacy, Part III
51mPrivate Boosting and Noise-tolerant Learning
17mData privacy and reliable learning
31mScaling privacy embedded distributed ML systems
33mPrivacy of Decentralized Machine Learning