Differentiable Programming in C++

on YouTube ↗59m

" summary

Claims that integrating automatic differentiation into the compiler yields concrete speed and compatibility wins: two implementations — Clad (AST-level) and Enzyme (LLVM IR) — produce gradients from real C++ and other front ends, Enzyme achieves about 4.2x speedup over a reference pipeline on a mixed ML/scientific suite, and compiler-level AD enables differentiating arbitrary GPU kernels and supports OpenMP and MPI. The approach avoids rewriting code, reduces maintenance, and is actively pushed toward standardization via SG19.

AI-written summary · about

" chapters (35)

1. 0:00Speakers
2. 0:15What is this talk about?
3. 0:25Outline
4. 2:29How fast he ran? What does that even mean?
5. 3:26Measuring the rate of change
6. 5:13Derivatives: measure the rate of change
7. 6:55The longer the distance the more parameters
8. 9:41Computing Derivatives
9. 11:26Numerical Differentiation
10. 14:28Automatic and Symbolic Differentiation
11. 16:40AD. Algorithm Decomposition
12. 17:42AD. Chain Rule
13. 19:14AD step-by-step. Forward Mode
14. 20:18AD step-by-step. Reverse Mode
15. 21:38AD Control Flow
16. 23:10AD. Cheap Gradient Principle
17. 23:42Uses of AD outside of Deep Learning
18. 24:23Deep Learning & Automatic Differentiation
19. 24:58Backpropagation
20. 27:17Differentiable Programming
21. 30:50C++ Automatic Differentiation Wish List
22. 32:42Existing AD Approaches (2/3)
23. 38:42Implementation of AD in Clang/LLVM
24. 40:03Case Study 1: Clad - AD of Clang AST
25. 43:00Clad Key Insights
26. 43:27Existing Automatic Differentiation Pipelines
27. 45:33Vector Normalization: LICM then AD
28. 46:20Vector Normalization: AD, then LICM
29. 47:17Optimization & Automatic Differentiation
30. 48:11Case Study 2: Enzyme - AD of LLVM IR
31. 49:27Enzyme Evaluation
32. 50:41Speedup of Enzyme
33. 53:12Key . Enzyme Insights
34. 54:12Overall AD Compiler Insights
35. 56:24Standardization Efforts

" related talks

• 39mDifferentiable Functional ProgrammingNoel Welsh · Scala Days New York 2018
• 15mCoarsening Optimization for Differentiable ProgrammingErik Meijer, Guoqiang Zhang, Xipeng Shen, Olin Shivers, Irene Dea, Samantha Andow, Emilio Arroyo-Fang, Neal Gafter, Johann George, Melissa Grueter, Steffi Stumpos, Alanna Tempest, Christy Warden, Shannon Yang · OOPSLA 2021
• 36mProgramming Languages for Automatic Differentiation: What Now?Damiano Mazza · LAFI 2022
• 16mAlgorithmic Differentiation: C++ & Extremum EstimationMatt P. Dziubinski · CppCon 2015
• 54mGeneralized Physics-Informed Learning through Language-Wide Differentiable ProgrammingChris Rackauckas · AAAI-MLPS 2020
• 26mAdding AD to Scheme by Differentiating the InterpreterBarak A. Pearlmutter, Mehrdad Maleki · Scheme 2021
• 05mPartial Evaluation of Forward-Mode Automatic DifferentiationDavid Broman, Viktor Palmkvist, Oscar Eriksson · LAFI 2023
• 29mPartial Evaluation of Automatic Differentiation for Differential-Algebraic Equati...David Broman, Viktor Palmkvist, Oscar Eriksson · GPCE 2023