Gemma 4 Deep Dive
summary
Concrete numbers and engineering choices decide whether to watch: 31B dense, 26B MoE (128 experts, 8 active) with a 256K context and interleaved local/global attention (sliding windows 1,024/512) claim top-six open‑source performance; PLE in flash with 256‑dim per‑layer embeddings enables effective 2B/4B on‑device models; grouped query attention (8 queries per KV) and a 305M conformer plus audio tokenizer add multimodal/audio support; Apache 2.0 licensing and self‑host paths (Hugging Face, Ollama) plus Vertex AI deployment round out practical adoption details.
AI-written summary · about
chapters (11)
- 0:00Start
- 0:28Introduction to the Gemma 4 model family and its four size categories
- 1:54Shift to Apache 2.0 licensing for developer accessibility
- 2:25Deep dive into the 31B dense reasoning and 26B mixture-of-experts (MoE) models
- 3:30Overview of on-device effective models (2B and 4B) with multimodal support
- 4:21Architectural updates: interleaved local/global attention and grouped query attention
- 6:51Explanation of the new MoE architecture (128 experts, 8 active)
- 7:44Implementation of Per Layer Embeddings (PLE) to optimize on-device memory
- 11:06Multimodal advances: variable aspect ratios and resolutions for vision encoders
- 16:31Audio processing enhancements via conformer architecture and audio tokenizers
- 18:07Getting started: self-hosting (Hugging Face, Ollama) and cloud deployment (Vertex AI)
related talks
16mGemma 4: Compacting Intelligence for the Edge
10mGemma 3 architecture and design
27mDeepdive into Gemma 3
14mGemma on mobile and web. Best and worst practices
10mDemo: Using Gemma with the Hugging Face ecosystem
39mFlutter Project with Gemma: A State-of-the-Art Open Model
07mWelcome to the Gemmaverse
40mBuilding amazing Front-End Project with Gemma: A State-of-the-Art Open Model