Revealing the Unseen: Visualizing Attention in Transformer Models
summary
Explains how tokenization and high‑dimensional embeddings feed into query, key, value projections and how dot‑product self‑attention produces context‑aware vectors; shows masked attention for generation vs. encoder only behavior; demonstrates multi‑head attention learning distinct roles and visualizes those roles with two tools called expert and birth Vis (works with Hugging Face models like GPT2 and BERT). Shows a concrete bias example where attention links “his”→doctor and “her”→nurse, and suggests fine‑tuning to correct it.
AI-written summary · about
chapters (13)
- 0:00Transformers and attention
- 1:58Language models process text
- 5:10Self-attention and embeddings
- 7:38Masked self-attention for generation
- 9:06Query, key, and value vectors
- 10:50Multi-head attention benefits
- 11:55Encoder and decoder blocks
- 13:20Visualizing attention tools
- 16:35Attention patterns in heads
- 17:36Testing pronoun context
- 18:42Detecting bias in attention
- 19:25Exploring attention with BERT VIS
- 22:13Summary
related talks
26mA Shallow Introduction to Self-Attention
01h 10mIntroduction to Transformers
45mKeys, Queries, and Values: The Celestial Mechanics of Attention
38mUnderstanding and Applying Self-Attention for NLP
01h 16mHow Do Transformers Work?
01h 25mPaper Discussion: Attention is all you need
26mRethinking the Transformer: Toward Native Multimodal Architectures
13mInside Multimodal Models