Lightning Talk: Exploring PiPPY, Tensor Parallel and Torchserve for Large...
summary
Covers how Torchserve runs large-model inference using PiPPY pipeline parallelism and tensor parallel APIs plus integrations with DeepSpeed, Hugging Face Accelerate, Neuron SDK and Inferentia2. Claims: 30B model needs ~60 GB in fp16 and a 70B Llama needs ~140 GB (≈8×80GB GPUs), so partitioning across devices is required; deferred initialization and checkpoint conversion are practical hurdles; micro-batching, continuous batching and token streaming responses reduce latency; an Inferentia2 example shows about 3x cheaper on Inferentia vs equivalent GPUs.
AI-written summary · about
chapters (16)
- 0:00Distributed LLM inference
- 0:41Model parallelization necessity
- 1:01Tensor vs pipeline parallelism
- 2:24OSS model parallelization
- 3:25PyTorch automatic partitioning
- 4:28Distributed inference challenges
- 5:30Checkpoint conversion for PyTorch
- 6:37TorchServe distributed inference
- 6:58Pipeline parallelism in TorchServe
- 7:38Tensor parallelism optimization
- 8:00Micro batching and continuous batching
- 8:43Streaming response API
- 9:49Real-world Inferentia 2 example
- 10:32Beyond parallelism optimizations
- 11:35TorchServe LLM roadmap
- 11:55Closing remarks
related talks
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
27mTips and tricks for distributed large model training
29mMulti-Accelerator PyTorch Serving With NxD Inference and vLLM
24mTraining large scale models using PyTorch
11mLightning Talk: Trinity Large - Torchtitan on 2000+ B300s
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
34mPanel Discussion: Training and Inference at Planet Scale
29mTrain billion-parameter models with model parallelism on Amazon SageMaker