PyTorch 2.0: Dynamic Shapes Support

on YouTube ↗11m

" summary

PyTorch 2.0 adds a symbolic shapes system that lets compiled graphs accept varying tensor sizes by propagating symbolic dimensions (implemented with sympy), adding runtime guards and control-flow ops, and integrating into the JIT/export path and torchinductor. Benchmarks on variable-length language-model generation show smoother latency than eager and padded static builds, higher throughput, and drop compilation repetitions from about five or six to a single compile. The system is implemented end-to-end and yields measurable performance and compile-time gains.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 0:57Execution Flow
3. 2:33Symbolic Shapes
4. 3:46Metadata
5. 4:36Control Flow
6. 5:10Specialization
7. 9:58Conclusion

" related talks

• 24mWhat's New for Dynamic Shapes in PyTorch 2.1Edward Yang · PyTorch Conference 2023
• 13mPyTorch 2.0Peng Wu · AI Infra @Scale 2023
• 28mOn TensorsNishant Sinha · The Fifth Elephant 2020
• 24mAdvancements in High-Performance AI/ML through PyTorch's Python CompilerSam Gross, Suraj Subramanian, Justin Jeffress · PyCon US 2023
• 36mPyTorch: Fast Differentiable Dynamic Graphs in PythonSoumith Chintala · Strange Loop 2017
• 31mPyTorch 2.1 - New DevelopmentsSupriya Rao · Scale By the Bay 2023
• 01h 02mThe Shapes of Multi-Dimensional ArraysVincent Reverdy · CppCon 2020
• 54mKeynote: PyTorch 2.1 Technical Deep DiveJoe Spisak, Peng Wu, Mergen Nachin, Mark Saroufim, Yanan Cao, Will Constable, Mario Lezcano · PyTorch Conference 2023