torch::deploy: Running eager PyTorch models in production
summary
torch::deploy beta runs eager PyTorch models inside a C++ inference service by hosting multiple independent Python subinterpreters in one process, sharing a single libtorch backend for zero-copy tensor passing and reduced memory overhead. It targets Linux x86_64 and arm64, avoids the Python GIL, requires no scripting or tracing, and supports loading pip/conda extensions from standard Python environments via a custom linker that isolates interpreter symbols while exposing libtorch globally.
AI-written summary · about
chapters (5)
- 0:00Introduction to torch deploy
- 1:03Core concepts and usage
- 3:06Benefits and features
- 4:17System architecture
- 5:39Beta features and conclusion
related talks
14mLightning Talk: The Fastest Path to Production: PyTorch Inference in Python
22mIncreasing the Impact of AI Through Better Software
10mExecuTorch on Microcontrollers: Deploying PyTorch To...
51mSpeeding up deployment of machine learning models
33mExecutorch: Enabling and Accelerating PyTorch Models on Edge Devices
14mFlexible Deployment of PyTorch Models on MCU-Class...
10mResearch to Production: PyTorch JIT/TorchScript Updates
09mTogether Goes Brrr: Threading Research & Production with Torch Compile