PaliGemma – Making Gemma 2 see by adding a vision encoder
summary
PaliGemma adds a SigLIP vision encoder to Gemma 2, converting images to tokens for the LLM and enabling zero-shot segmentation and detection by encoding bounding boxes and discrete mask tokens plus a 4×4→64×64 mask upscaler. It is pre-trained on captioning, VQA, OCR, detection and segmentation with about 1 billion examples, then a short high-resolution stage; three model sizes and three input resolutions scale compute by 155x. Best results come from fine-tuning on task data; reported SOTA on table extraction, chest X‑ray reports, molecular and music OCR; also used for robotics with action tokens.
AI-written summary · about
chapters (12)
- 0:00PaliGemma intro
- 0:42Gemma vision with PaliGemma
- 1:26Zero-shot detection & segmentation
- 1:48Vision-language model integration
- 2:51Model sizes & resolution scaling
- 3:34Training for detection & segmentation
- 4:39Fine-tuning PaliGemma
- 5:20Performance benchmarks
- 6:47Real-world Gemmaverse apps
- 7:31Pi Zero robotics model
- 8:34PaliGemma 2 vs Gemma 3
- 9:41Summary & resources
related talks
10mDemo: Using Gemma with the Hugging Face ecosystem
10mAI for everyone with Gemma
15mGemma, DeepMind's Family of Open Models
12mGemma: The responsible way to build
16mOpen models in the Gemini era
08mDemo: Gemma on-device with MediaPipe and TensorFlow Lite
39mFlutter Project with Gemma: A State-of-the-Art Open Model
19mGemini 2.0: Beyond the Basics