Jeff Johnson, Research Engineer, Facebook

on YouTube ↗19m

" summary

Argues that many deep learning layers are too small for GPUs, shows an open-source Torch convolution engine (FB FFT) that reorders work to fit registers and beats cuDNN on 5x5 kernels, explains why typical conv → GEMM lowering fails on 3x3 feature maps and small minibatches, measures that modern networks are mostly bandwidth-bound not FLOPS-bound, and demonstrates that aggressive kernel fusion and data-layout choices reduce memory traffic and latency and materially speed training.

AI-written summary · about

" chapters (24)

1. 0:00Introduction
2. 0:22Convolutional Neural Networks
3. 1:41Internal Networks
4. 2:35Image Classification
5. 3:21convolutional networks
6. 4:02training
7. 5:27deep numbers
8. 6:30more parameters
9. 7:28latency
10. 8:09small problems
11. 9:39vector processors
12. 10:38single instruction multiple thread
13. 11:02one bit processors
14. 11:36GPU vs CPU
15. 12:35Paralyzation
16. 13:40CPU vs GPU
17. 14:26Decomp
18. 15:10Nvidia
19. 16:01Research
20. 16:58Focus
21. 17:43Laziness
22. 18:09Fusion
23. 18:39CPU
24. 18:54Cradle Fusion

" related talks

• 57mDeep Learning with C++Peter Goldsborough · Meeting C++ 2017
• 30mUnderstanding GPUsVarun Nayyar · PyCon AU 2019
• 24mKeynote: Next-generation frameworks for Large-scale AIAnima Anandkumar · Scale By the Bay 2020
• 28mNext-generation frameworks for Large-scale Machine LearningAnima Anandkuma · Scale By The Bay 2019
• 55mA Tour of Deep Learning With C++Peter Goldsborough · CppCon 2017
• 58mDeep Learning in Real LifeYoann Benoit, Sandra Pietrowska · XebiCon'17
• 49mDistributed Deep LearningJim Dowling · Jfokus 2019
• 21mCe Zhang, Postdoctoral Researcher, Stanford UniversityCe Zhang · MLconf ATL 2015