Bridging Pixels and Words: The Transformer Vision Revolution
summary
Claims that ViTs work once images are broken into patches and that 60x60 patches yield ~700 tokens for a 512×512 RGB image, and that Vision Transformer (patches → ~700 tokens) enables global attention but needs huge data. Practical wins include Swin: ~10x faster via local window attention and hybrids; alignment uses CLIP contrastive alignment; LAVa pairs ViT + CLIP + fine‑tuned LLaMA and beats closed models. Known failures: state‑of‑the‑art VLMs flunk 2D game reasoning; screenshots as visual tokens compress long text to 64 tokens with ~3% loss.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:44Why vision matters
- 7:4125 years of computer vision
- 15:08From CNNs to transformers
- 17:38Transformers for pixels
- 25:00Vision language models
- 32:50VLM limitations and opportunities
- 39:27Real-world applications
related talks
33mSeeing Through the Lens of Transformers: A New Era in Computer Vision
17mHow Transformers Finally Ate Vision
43mVisual Attention is All You Need: Introduction to Visual Transformers
26mRethinking the Transformer: Toward Native Multimodal Architectures
28mLarge Multimodal Models: Understanding the State of the Art in Compute
35mUnpacking Scattering Vision Transformer: A New Dawn in Computer Vision
34mBeyond Transformers - Intro to RWKV Architecture & The World To...
17mVision AI in 2025