Getting started with automatic differentiation
summary
Explains how automatic differentiation computes exact derivatives and scales to millions of parameters, demonstrates gradient computation and repeated grad nesting to get higher derivatives, and uses jax, TensorFlow, and PyTorch for concrete examples. Builds a spiral illustration and draws its derivative outward as computational art spiral derivative, runs gradient descent that converges to local minima (and sometimes bounces with a large step), and extracts edges from a grayscale photo of Pete as edge detection on Pete's photo.
AI-written summary · about
chapters (8)
- 0:00Intro
- 1:43WRITING A NUMERIC PROGRAM
- 3:06RATE OF CHANGE AS A SLOPE
- 5:12AUTOMATIC DIFFERENTIATION IN PYTHON
- 9:10PLOTTING DERIVATIVES
- 13:27EDGES IN IMAGES
- 14:47OPTIMIZATION WITH JAX
- 18:18GRADIENT DESCENT
related talks
39mDifferentiable Functional Programming
46mGradient Descent: The Ultimate Optimizer
51mDeep Learning - What is it and What It Can Do For You
16mAlgorithmic Differentiation: C++ & Extremum Estimation
35mDerivatives - Important Concept. Simple to grasp in Kotlin
29mGradient descent is taking away your Android developer job
34mDifferentiable Functional Programming
41mOptimization Algorithms for Deep Learning