How Transformers Finally Ate Vision
summary
Explains a concrete claim: massive ViT-specific pretraining plus LLM-derived speedups make plain transformers beat conv nets in vision. Cites ViT, Swin, ConvNeXt, Hiera, MAE and DINOv2/3 as the actual tools, shows MAE pretraining produces rich frozen features, notes flash attention removes the n^4 speed barrier, and reports deployment results: 800M parameters SAM 3 at ~300 ms on a T4 versus RF100VL fine-tuning that yields a 40x speedup (or 15x for smaller gains) for comparable accuracy.
AI-written summary · about
chapters (20)
- 0:00Transformers in vision
- 1:18Convolutional neural networks
- 1:39Transformers' inductive bias
- 2:20Vision Transformers challenges
- 3:02Why Vision Transformers outperformed CNNs
- 3:42Evolution: Swin and ConvNeXt
- 5:28ConvNeXt: borrowing transformer ideas into CNNs
- 6:31ConvNeXt speed limits
- 7:15Masked Autoencoders
- 8:17DINOv2 self-supervised
- 9:19Self-supervised vs supervised
- 9:41Flash Attention speedups
- 10:43SAM foundation model
- 11:48Deploying large models
- 12:54RF100VL dataset
- 13:37Neural architecture search
- 14:21RFDetR real-time
- 15:03Summary: ViT, LLM speedups, NAS
- 15:24Future: video, multimodal, JEPA
- 16:28Q&A
related talks
33mSeeing Through the Lens of Transformers: A New Era in Computer Vision
41mBridging Pixels and Words: The Transformer Vision Revolution
17mVision AI in 2025
43mVisual Attention is All You Need: Introduction to Visual Transformers
01h 16mHow Do Transformers Work?
35mUnpacking Scattering Vision Transformer: A New Dawn in Computer Vision
23mUnveiling the Journey of NLP
26mRethinking the Transformer: Toward Native Multimodal Architectures