Long-context Attention in Near-Linear Time
summary
HyperAttention claims near-linear time attention by sampling columns and rows based on two fine-grained parameters: the max column norm of the row-normalized softmax and the post-pruning ratio of row norms of the unnormalized exponential matrix, yielding an operator-norm guarantee when those are small. The system uses LSH to find heavy entries, plugs into FlashAttention, and shows concrete gains: 50% faster inference for ChatGLM2 at 32k contexts with perplexity rising 5.6→6.3 and a 5× single-layer speedup at 131k.
AI-written summary · about
chapters (10)
- 0:00Introduction to Transformers
- 2:09Transformer architecture
- 3:48The self-attention bottleneck
- 5:59Attention mechanism internals
- 11:01Approximation strategies
- 14:37Linear time approximation
- 23:26Assumptions and bounds
- 31:46GPU-friendly LSH implementation
- 35:48Causal masking and experiments
- 37:02Performance and Q&A
related talks
01h 30mToward Understanding In-context Learning
22mCost-Efficient Large Language Model Serving for Multi-turn Conversations with...
01h 25mPaper Discussion: Attention is all you need
34mAttention is not all you need - Keynote
38mUnderstanding and Applying Self-Attention for NLP
48mThe Magic Of attention in LLMS
51mLong context modeling and generalization: two perspectives
24mRevealing the Unseen: Visualizing Attention in Transformer Models