Differentiable Functional Programming

on YouTube ↗39m

" summary

Deep learning is framed as supervised learning of parameterized functions optimized by gradient descent, and automatic differentiation is implemented concretely: forward mode via dual numbers and a flatMap monad, reverse mode via continuations so storing a tape lets you run gradients backward; Scala prototypes compile to GPU, code lives on GitHub, and examples (sine^2) give identical derivatives. Reverse-mode AD is preferred when outputs are small, and a toy differentiable system is claimed doable in a day or two; experimental projects include Nexus and reverse-mode AD in language forks.

AI-written summary · about

" chapters (27)

1. 0:00Intro
2. 2:24Parametrised functions Supervised learning Gradient descent
3. 4:04Choose parameters to fit data
4. 5:20Algorithmically choose parameters
5. 6:50Calculate the derivative of the loss function with respect to the parameters
6. 7:31Calculate gradient for current parameters
7. 8:43Deep learning is supervised learning of parameterised functions by gradient descent
8. 9:49Tensor multiplication and non-linearity
9. 10:01Tensors: multidimensional arrays
10. 13:26Conditionals and loops
11. 14:47Algorithms for calculating gradients
12. 15:08Composition of Derivatives
13. 17:17Symbolic differentiation
14. 17:46No loops or conditionals
15. 18:39Inexact. Choosing e is hard
16. 19:05Two main algorithms: forward and reverse
17. 19:26Calculate with dual numbers
18. 25:33There is a monad for Dual
19. 26:20Forward-mode scales in the size of the input dimension
20. 27:02Chain rule doesn't care about order
21. 28:26Use a monad (or continuations)
22. 29:43Reverse-mode scales in the size of the output dimension
23. 32:27Tensor dimensions must agree
24. 33:31Solution: expressive type systems
25. 34:35Linear logic and differential A-calculus
26. 35:34Need compilation (to GPU) for performance
27. 37:34Scala is well positioned for the next generation of DL systems

" related talks

• 34mDifferentiable Functional ProgrammingNoel Welsh · Scala Days Berlin 2018
• 36mProgramming Languages for Automatic Differentiation: What Now?Damiano Mazza · LAFI 2022
• 59mDifferentiable Programming in C++Vassil Vassilev, William Moses · CppCon 2021
• 35mDerivatives - Important Concept. Simple to grasp in KotlinBreandan Considine · KotlinConf 2019
• 26mAdding AD to Scheme by Differentiating the InterpreterBarak A. Pearlmutter, Mehrdad Maleki · Scheme 2021
• 54mGeneralized Physics-Informed Learning through Language-Wide Differentiable ProgrammingChris Rackauckas · AAAI-MLPS 2020
• 29mPartial Evaluation of Automatic Differentiation for Differential-Algebraic Equati...David Broman, Viktor Palmkvist, Oscar Eriksson · GPCE 2023
• 14mNominal Semantics for First-class Automatic DifferentiationJack Czenszak, Alexander K. Lew · LAFI 2026