Talk by Amer Sinha (Google)
summary
A one billion parameter Gemma variant is trained from scratch with DP-SGD, yielding epsilon < 2 sequence-level DP on 1,024-token units packed from web sources and released as open weights. It trains on 13 trillion tokens with very large batches (order 500k) and trades much more compute (roughly 100x versus non-private) for utility, matching GPT-2–era pretraining but showing low uniform-sample memorization while targeted extraction finds ~7% recovery of PII due to deduplication and privacy-unit limits.
AI-written summary · about
chapters (12)
- 0:00VaultGemma intro and privacy
- 2:09Why privacy matters for GenAI
- 3:12DP-SGD for model training
- 5:02VaultGemma architecture and training
- 8:45Privacy unit challenges and DP
- 11:37Motivations for VaultGemma
- 12:20Performance comparison
- 13:41Cost of privacy in compute
- 19:08Memorization analysis with DP
- 26:40DP training scaling laws
- 31:01Key VaultGemma findings
- 33:14Conclusion and future work
related talks
17mPrivate Tuning of LLMs in Practice: From VaultGemma to Custom Fine-Tuning
41mInside GPT – The Maths Behind the Magic
10mDemo: Building a Gemma chatbot with Keras
09mUsing and fine-tuning Gemma 3
17mUnlock the power of Generative AI: From understanding its principles to utilizing its power
09mOpen models and open compression
14mToward Provably Private Insights into AI Use
19mGemma 4 Deep Dive