Gemma 3 architecture and design
summary
Gemma 3 uses a 256,000-token tokenizer updated for 64 languages, improving multilingual and finetune coverage; models come in 1B, 2B, 4B, 12B, 27B with predictable scaling and identical training/data mixes so capacity swaps produce no surprises. Long context extends from 8k to 128k tokens (12B/27B work to ~256k and 27B can be probed toward 500k), achieved by tuning two global-attention parameters and adding a scale factor. Runtime saves come from five local attention layers per global layer, cutting a 12B inference memory need to ~26 GB versus ~49 GB for Llama-style models.
AI-written summary · about
related talks
19mGemma 4 Deep Dive
18mImproving open language models at a practical size
16mOpen models in the Gemini era
10mDemo: Using Gemma with the Hugging Face ecosystem
10mAI for everyone with Gemma
15mGemma, DeepMind's Family of Open Models
54mGemini, Google's Large Language Model
39mFlutter Project with Gemma: A State-of-the-Art Open Model