Fast deep recurrent net training
summary
Deep Speech is a convolutional-to-biRNN speech recognizer trained with CTC; a seven-layer bidirectional baseline gets stuck at a training glass ceiling. Practical fixes that work here: serve clips by increasing length in epoch one (sorted grad) because CTC roughly scales with audio length, add batch normalization on vertical connections, and swap to lightweight GRU cells. Implemented in C++/CUDA/MPI, these changes speed convergence, reduce training instability, and reach lower training cost minima.
AI-written summary · about
chapters (10)
- 0:00Introduction and motivation
- 3:42What is Deep Speech
- 5:52Making model deeper to reduce bias
- 7:35SortaGrad optimization trick
- 10:04Batch normalization benefits
- 12:31GRU cells vs vanilla RNNs
- 16:29Optimization before scaling
- 19:27Q&A: Bias vs optimization
- 20:30Q&A: Optimizers beyond SGD
- 22:01Q&A: Optimizers impact on training
related talks
45mThe Deep Learning Revolution in Automatic Speech Recognition
27mThe Story of End to End Models in Deep Learning
37mWhat's new in Deep Learning?
28mHow to use Deep Learning to process audio data and classify sounds.
59mDeep Neural Networks for Video Applications
47mInterpreting Deep Neural Networks (DNNs)
06mDeepRange: Acoustic Ranging via Deep Learning
26mUsing deep learning in natural language processing