Attention and Memory in Deep Learning Networks
summary
Shows that attention removes the encoder bottleneck in sequence‑to‑sequence translation so sentence length (a 38‑word example) stops collapsing accuracy after ~10–20 words, and that simple fixes to RNNs—locked (variational) dropout and zoneout—reduce overfitting with one‑line changes in Keras. Demonstrates pointer networks that exactly recover convex hulls and nearly solve small TSPs, a dynamic memory/co‑attention approach that locates SQuAD spans, and a sentinel mixture that combines pointing with vocab generation; Facebook sees big gains when attention looks back over 2,000 words.
AI-written summary · about
chapters (10)
- 0:00Start
- 7:42The limits of restricted memory
- 9:57Hidden state is expensive A simple recommendation may be why not increase the capacity of our hidden state?
- 13:27Regularizing RNN Zoneout (Krueger et al. 2016) Offers two main advantages over locked dropout
- 14:17Translation as an example Stunningly you can supply English as input, German as expected output, and the model learns to translate
- 15:02Translation as an example Stunningly you can supply English as input, German as expected output, and the model learns to translate
- 17:09Translation as an example As we process each word on the decoder side, we query the source encoding for relevant information
- 23:52The DMN's input modules
- 29:57Attention Sum (Kadlec et al. 2016)
- 34:03Pointer Sentinel (Merity et al. 2016)
related talks
01h 25mPaper Discussion: Attention is all you need
50mSequential Attention-Based Neural Machine Translation
26mA Shallow Introduction to Self-Attention
20mMelanie Warrick, Deep Learning Engineer, Skymind.io
53mTensorflow, deep learning and modern RNN architectures, without a PhD
38mUnderstanding and Applying Self-Attention for NLP
26mUsing deep learning in natural language processing
36mNeural Networks