Differentiable Functional Programming
summary
Claim: deep learning is mostly function composition plus differentiation and is straightforward to implement computationally — a toy system can be built in days. Implements exact gradients with forward-mode dual numbers (easy, scales with input dimension) and reverse-mode via monads/continuations (preferred in deep nets, scales with output dimension). Shows Scala examples using Spire’s "jet" duals, reproduces matching gradients, and points to projects that compile/typesafe tensors: Nexus typed tensors and a staging approach (lanten) to emit GPU code. Typical networks run 30–60 layers.
AI-written summary · about
chapters (16)
- 0:00Intro
- 2:56Parametrised functions Supervised learning Gradient descent
- 7:48Calculate gradient for current parameters
- 9:19Deep learning is supervised learning of parameterised functions by gradient descent
- 10:13Tensor multiplication and non-linearity
- 15:25Algorithms for calculating gradients
- 15:36Composition of Derivatives
- 17:06Mathematician's approach
- 18:25Programmer's approach
- 19:26Automatic differentiation approach
- 20:07Calculate with dual numbers
- 24:37Forward-mode scales in the size of the input dimension
- 25:12Chain rule doesn't care about order
- 30:13Tensor dimensions must agree
- 31:33Solution: expressive type systems
- 32:11Need compilation (to GPU) for performance
related talks
39mDifferentiable Functional Programming
37mMonadic Deep Learning
37mNeural Networks are Programs, Too
35mDerivatives - Important Concept. Simple to grasp in Kotlin
36mProgramming Languages for Automatic Differentiation: What Now?
51mDeep Learning - What is it and What It Can Do For You
17mData-Parallel Differentiation by Optic Composition
35mTowards Typesafe Deep Learning in Scala