Theoretical and Practical Insights from Linear Transformers
summary
One-layer linear transformers that minimize the in‑context loss provably reproduce one-step gradient descent (under Gaussian x and θ assumptions), and multilayer versions have stationary points whose layerwise outputs match L steps of gradient descent with a learned preconditioned by covariance inverse update; proofs use constructions on linear attention and experiments recover the same key–query–value products, show linear attention outperforms softmax on this task, and report Adam much faster than SGD in optimization.
AI-written summary · about