The Best We Can Do?
summary
Claim: compiling high-level code to a graph and doing local graph reduction yields massive parallelism. Prototype compiles to HML, stores programs as graphs, runs a Rust kernel on CUDA GPUs and reports 73 billion graph operations per second on an Nvidia card. It discovers parallelism automatically, fuses maps, and eliminates stacks, but it has limits: restricted recursion patterns, 29‑bit integer packing for peak CUDA throughput, and non‑deterministic IO ordering. The project is called Higher Order Virtual Machine.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:46Lambda calculus basics
- 6:13Lambda calculus limits
- 8:01Graph reduction intro
- 17:04Graph vs Lambda expressions
- 19:33Higher Order Virtual Machine implementation details
- 21:44Automatic parallelism
- 26:51Compile-time optimizations
- 28:19No program stack
- 31:34Graph reduction implications
- 34:11Optimality and garbage collection
- 38:22Runtime execution
- 42:15Compile-time vs runtime
- 44:39Lazy evaluation future
- 46:10Limitations and constraints
- 48:01Practical applications
- 50:28Unsolved challenges
- 51:14Future of programming
- 52:44Q&A
related talks
33mParallel Processors: Past & Future Connections Between LLMs and OS Kernels.
28mOptimizing Large-Scale Model Training with Ray Compiled Graphs
26mEfficiently serving LLMs at scale.
40mEthereum at Hyperscale
41mHelion's New CuTe DSL and Pallas Backends and How We Built Them Via Agents
30mWrite One Run Anywhere, But for GPUs
01h 03mKeynote: Why Functional Programming Matters
08mGPU Programming with Triton Kernels