Succinct Neural Networks
summary
Proposes a DNN architecture that is succinct meaning a single constant-size neural network S is convolved and recurrently applied to simulate the tableau of any constant-size Turing machine given enough layers T, and thus can represent any constant-state algorithm. Architecture combines recurrent + convolutional modules and an extra writable layer to store training examples; proof shows random initializations (or SGD variants) suffice to match the best efficient learning algorithm on average, but gradients explode, no practical implementation yet.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:52Learning Algorithms
- 1:37Folk Theorem for Learning
- 4:31Succinct Neural Networks
- 6:45Turing Machines
- 9:45Small Behavior
- 11:28Training Algorithms
- 15:05Learning
- 19:28Hidden constants
- 22:23Gradient descent
- 25:27Numerical stability
- 28:26Random circuits
- 29:57Theorem
- 32:40Model of Learning
- 38:05Stochastic Gradient Descent
related talks
32mA word is worth a thousand pictures: Convolutional methods for text
19mAbstract Neural Networks
29mCompositional sparsity and learnability
47mCapacity of Neural Networks for Lifelong Learning of Composable Tasks
55mNetworks of Spiking Neurons Learn to Learn and Remember
21mNeural Turing Machines: Perils and Promise
39mComputational Thinking on Learning Models
12mComputation with sequences of neuronal assemblies