Scheme as a framework for Deep Learning
summary
Claims a compiler-integrated AD system that uses abstract interpretation/partial evaluation and CPS-based checkpointing to move reflective source-to-source differentiation to compile time, producing tight code via inlining and dead‑code elimination. Implements primops for interrupt/resume (capsules) so divide‑and‑conquer checkpointing reduces reverse-mode space from O(w·t) to O(w·log t) at O(t·log t) time. Prototype language Scorch lifts primitives to tensors, runs ResNet to quarter‑million layers with linear scaling, and reports forward 28 lines / reverse 155 lines for AD transforms.
AI-written summary · about
chapters (8)
- 0:00Start
- 8:26Migrate Reflective Source To Source Transformation from Runtime To Compile Time with Abstract Interpretation
- 23:15Reverse Mode Transformation
- 23:36How To Do Reverse Mode
- 26:57Probabilistic Programming
- 32:47History of Backpropagation
- 47:46Reference Implementation
- 51:20Ray Tracer
related talks
36mProgramming Languages for Automatic Differentiation: What Now?
28mFunctional Sparse Tensor Compilation
29mStaging Automatic Differentiation with Fusion
39mDifferentiable Functional Programming
56mAutomatically Scalable Computation
59mDifferentiable Programming in C++
54mProgramming Language Design and Implementation for the Machine Learning Era: A Personal(…)
25mLuminal - Search-Based Deep Learning Compilers