Do transformers need all that attention?
summary
Claims that many Transformer layers are redundant and can be removed with minimal loss: using Attention Drop and a redundancy metric based on cosine similarity the team removes up to ~50% of attention layers (e.g., cutting 80→40) and reports roughly half the memory usage and ~99% of original accuracy on examples like Llama‑2/Mistral‑7B after removing eight attention layers. Removing MLP layers alone hurts performance, while Joint Layer Drop (prune attention then MLP) yields the best memory and inference‑speed tradeoffs.
AI-written summary · about
chapters (7)
- 0:00Introduction and thanks
- 1:26What are transformers
- 4:23Attention layer drops
- 11:32Attention drop vs block drop
- 16:54Benefits of attention drops
- 22:20Key findings summary
- 23:02Q&A session
related talks
15mBreaking the Layer Barrier: Remodeling Private Transformer Inference with...
26mRethinking the Transformer: Toward Native Multimodal Architectures
01h 16mHow Do Transformers Work?
40mAn Occam's Razor Principle for Transformers?
01h 10mIntroduction to Transformers
01h 25mPaper Discussion: Attention is all you need
38mHow Transformers Learn Causal Structure with Gradient Descent
38mUnderstanding and Applying Self-Attention for NLP