Parallel PyTorch Inference with Python Free-Threading
summary
A concrete claim: building and running a Python 3.13-T free-threaded build lets multiple Python threads run PyTorch inference truly in parallel. The demo trains a 124M-parameter GPT-2 from scratch and shows a threaded image-decoding benchmark where threads scale while processes saturate because of CUDA context switching. A threaded HTTP server for token-by-token LLM serving yields much higher throughput and lower latency overall, though single-thread performance can be slightly slower and many C/C++ bindings must be rebuilt and tested.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:07Python’s GIL
- 5:24Free-threading removal
- 9:25Free-threaded ecosystem challenges
- 15:52Parallel data processing
- 22:58LLM inference with HTTP server
- 31:59Resources
related talks
18mHigh Performance LLM Inference in Pure Python with PyTorch Custom Ops
23mHow Pytorch Powers Training Inference
28mUnlocking Smarter AI with Inference-Time Scaling
27mOptimizing PyTorch on CPU-GPU Coherent Platforms
23mTorchTPU: Expanding TPU Programmabil...
16mDe-mystifying PyTorch for ASICs: When (and Why) To Move Your Development To AI A...
09mTogether Goes Brrr: Threading Research & Production with Torch Compile
23mThe Race to the Bottom: Low Latency in the age of the Transformer