Using deep learning in natural language processing
summary
Describes Google Translate moving from phrase‑based to a seq2seq recurrent neural network with attention trained across about 26 languages, using data and model parallelism (encoder/decoder split and 8 stacked RNN layers). Implementation detail: training uses floating‑point backprop but deployment quantizes states to 8‑bit and some matrices to 16‑bit, keeping perplexity essentially unchanged while giving large runtime speedups on TPU; GPUs can be slower than CPUs for production decoding due to I/O.
AI-written summary · about
chapters (7)
- 0:00NLP and deep learning intro
- 3:38Why NLP matters
- 10:56Recurrent neural networks for language
- 14:28Seq2seq translation models
- 17:40Training with gradient descent
- 21:57Fast inference via quantization
- 25:09Summary
related talks
39mDeep Learning for Machine Translation: a paradigm shift
24mQuoc Le, Software Engineer, Google
26mQuoc Le, Software Engineer, Google
53mTensorflow, deep learning and modern RNN architectures, without a PhD
01h 25mPaper Discussion: Attention is all you need
07mMachine learning at Google
01h 59mDeep Learning and Modern NLP
42mAttention and Memory in Deep Learning Networks