Parallelism-preserving automatic differentiation for second-order array languages
summary
Paper gives concrete AD rules for map and scan in a second-order array DSL, implements forward-mode JVP and proves map-differentiation is componentwise; scan uses a concrete scan pairing trick (pair values with sensitivities) to restore associativity and run JVP. Reverse-mode VJP is implemented but turns out very complex and can blow up asymptotically, violating the cheap‑gradient principle. Experiments target Dex-like pipelines; Enzyme and JAX are cited as lower‑level or unrolling alternatives.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 7:40Ad systems
- 11:03Implementation
- 12:55Associative law
- 15:50Gradientbased optimization
- 18:48Cheap gradient principle
- 19:56Next steps
- 21:15What if
- 22:44Summary
- 23:53Defunctionalization
related talks
29mPartial Evaluation of Automatic Differentiation for Differential-Algebraic Equati...
11mAbstract Interpretation for Automatic Differentiation
36mProgramming Languages for Automatic Differentiation: What Now?
05mPartial Evaluation of Forward-Mode Automatic Differentiation
26mAdding AD to Scheme by Differentiating the Interpreter
01h 21mNon-local compiler transformations in the presence of dynamic dispatch
29mStaging Automatic Differentiation with Fusion
25mYou Only Linearize Once: Tangents Transpose to Gradients