Lightning Talk: CUDAGraph in a Partial Graph World
summary
CUDAGraph trees integrate with torch.compile to replace per-graph private pools with a single CUDA memory pool that recreates eager allocation sequences, eliminating repeated copies between captures and cutting memory use in benchmarks on A100s while still improving throughput especially at batch size one. The system specializes on tensor liveness and control-flow hot paths, building a tree of captures with warm-up recordings and fast re-records per unique shape so Dynamic shapes work by re-recording. Known limits include no eager-pool sharing and 65 KB per kernel launch driver overhead.
AI-written summary · about
chapters (8)
- 0:00Introduction to cographs
- 1:24Cograph limitations
- 2:28Memory challenges
- 4:21New cograph trees feature
- 6:42Advanced usage constraints
- 7:18Use cases and examples
- 8:35Performance results
- 9:34Conclusion and Q&A
related talks
26mParameterized CUDA Graph Launch in PyTorch: CUDA Graphs Without the Pain
36mPyTorch: Fast Differentiable Dynamic Graphs in Python
25m"Rust for AI & Accelerated Computing"
14mP3: Distributed Deep Graph Learning at Scale
08mPyTorch 2.0: TorchDynamo
05mLightning Talk: GPU Programming with C++ and Triton
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
15mGraCE: Unlocking CUDA Graphs with Compiler Support for ML Workloads