Unlocking Parallel PyTorch Inference (and More!)
summary
Claim: PyTorch inference runs concurrently inside the same process using Python free-threading. Built a multi-threaded pure-Python Paralleledia web stack and a streaming async.io HTTP server that yields tokens, hosts a 2015 Wikipedia dump in-process, and shows parallel generators working under load (14 threads). Measured 70 vs 7.35 requests/sec and tail latency ~350ms vs 1.75s with free-threading; a local GPT-2 trained ~19,072 steps over 3.5 days. It works end-to-end but skips TorchDynamo, KV cache, batching and needs Triton/VLM for production performance.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:03Python free threading overview
- 3:52Python GIL history
- 9:31Python 3.13 free threading
- 11:16PyTorch inference crash course
- 15:09Testing PyTorch inference
- 16:52Paralleledia server
- 22:35PyTorch free threading limits
- 24:39Real-world use cases
- 26:29Wikipedia in single process
- 33:10Resources
- 34:15Q&A
related talks
39mUnlocking the Parallel Universe: Subinterpreters and Free-Threading in...
43mExploring Deep Learning Framework PyTorch
30mSpeed Up Your Data Processing
30mPython Serialization and the Hallows of Parallel Computing
30mParallel computing in Python: Current state and recent advances
02h 57mPython Concurrency: from beginner to pro
19mIncreasing the Impact of AI Through Better Software (Pytorch Updates)
01h 01mParallel Python Analyzing Large Data Sets