Lightning Talk: The Fastest Path to Production: PyTorch Inference in Python
summary
Describes a concrete path to run PyTorch in production using torch.compile with new APIs to cache compilation artifacts across devices and python bindings for AOT Inductor, plus using torchserve with backend Python workers. Recommends dynamic batching (example: batch=4, wait=30ms), CUDA graphs to cut overhead, and py‑profiler to spot many tiny GPU kernels. Warns zipping LLaMA‑7B takes 24 min (unzip 3 min) and that mmap + meta‑device init yields ~10x faster load; CPU thread migration can cause 5–10x slowdowns, fixed by core affinity and tuned torch.set_num_threads.
AI-written summary · about
chapters (10)
- 0:00Fast PyTorch inference intro
- 1:46TorchServe vs FastAPI
- 3:28Dynamic batching optimization
- 5:12Debugging slow inference
- 6:35Latency reduction with TorchCompile
- 7:38Caching and warm starts
- 8:40Faster model loading
- 9:24CPU thread affinity
- 10:48Cost savings from optimizations
- 11:49TorchServe for production
related talks
23mBlazing Fast GenAI Inference With Torch.compile
08mSponsored Session: Lightning Talk: PyTorch in Production: Boosting LLM Training and...
36mDL Compiler Panel Discussion
18mHigh Performance LLM Inference in Pure Python with PyTorch Custom Ops
14mLightning Talk: Accelerating PyTorch Models With Torch.compile's C++ Wrapper Mode
24mAdvancements in High-Performance AI/ML through PyTorch's Python Compiler
09mTogether Goes Brrr: Threading Research & Production with Torch Compile
32mKeynote: Fireside Chat with Soumith Chintala, Co-Creator of PyTorch & Merve Noyan