Random Embeddings, Matrix-valued Kernels and Deep Learning

on YouTube ↗56m

" summary

Claim: large-scale randomized kernel models match deep nets on a speech benchmark when scaled and optimized. Implementation uses random Fourier features (up to ~80k features), an MPI + shared-memory distributed ADMM solver on 256-node Blue Gene/Q, and LibSkylark software; training runs in hours and yields phoneme/frame error comparable to tuned DNNs (kernels gave the best published kernel phoneme error on that feature set). Quasi-Monte Carlo and a box discrepancy objective reduce feature count and improve gram-matrix reconstruction.

AI-written summary · about

" chapters (24)

1. 0:00Intro
2. 0:38Acknowledgements
3. 1:38Setting
4. 6:48Deep Learning is Supercharging Machine Learning
5. 8:51Machine Learning in 1990s
6. 12:03Kernel Methods and Neural Networks (Pre Google)
7. 14:05The Mathematical Naturalness of Kernel Methods
8. 16:11Scalability Challenges for Kemal Methods
9. 17:27Randomized Algorithms
10. 19:56Random Fourier Features (Rahimi & Recht, 2007)
11. 21:08DNN vs Kernel Methods on TIMIT (Speech)
12. 24:45DNNs vs Kernel Methods on TIMIT (Speech)
13. 30:45Distributed Convex Optimization
14. 32:58Distributed Block-splitting ADMM
15. 35:20Randomized Kernel methods on thousands of cores
16. 36:01Comparisons on MNIST
17. 37:51Efficiency of Random Embeddings
18. 38:45Quasi-Monte Carlo Sequences: Intuition
19. 42:21How do standard GMC sequences perform?
20. 45:25Does Box discrepancy explain behaviour of QMC
21. 45:40Learning Adaptive QMC Sequences
22. 46:22Randomization vs Optimization
23. 47:55Deep Learning with Kernels?
24. 48:36Conclusions

" related talks

• 01h 04mRandomized Embeddings and Neural NetworksMert Pilanci · Sketching and Algorithm Design 2023
• 43mThe Linear Algebra of Deep LearningElizabeth Ramírez · Code Mesh LDN 2020
• 32mHow Randomized Analysis Can Help Us Do “Data Science:” Examples in Deep Learning and Graph AnalysisFred Roosta · Randomized Numerical Linear Algebra and Applications 2018
• 01h 20mThe elusive generalization: classical bounds to double descent to grokkingMisha Belkin · Modern Paradigms in Generalization Boot Camp 2024
• 33mThe Power and Limitations of Kernel LearningMisha Belkin · Optimization, Statistics and Uncertainty
• 01h 30mIntroduction to Deep Learning & Natural Language ProcessingRaghotham Sripadraj, Nischal HP · PyData London 2016
• 20mOPER: Optimality-Guided Embedding Table Parallelization for Large-scale...Boyuan Feng, Yufei Ding, Zheng Wang, Yuke Wang, Guyue Huang, Ang Li, Dheevatsa Mudigere, Bharath Muthiah · USENIX ATC 2024
• 28mRandomized Constrained Matrix DecompositionsBenjamin Erichson · Randomized Numerical Linear Algebra and Applications 2018