Keynote: How to Leverage PyTorch to Scale AI Training and Inferencing
summary
IBM shows concrete scaling steps using FSDP, adding a rate‑limiter API, S3 distributed checkpointing, and hybrid sharding to cut network traffic. Training on 256 GPUs with 400 Gbps yields ~3,000 tokens/sec/GPU for a 7B LLaMA (≈70B tokens/day), ~1,600 tps/GPU for 13B, and ~300 tps/GPU for early 70B runs. Inference combines PyTorch compile, accelerated/flash attention and tensor parallelism to hit about 29–30 ms per token on a 70B model (benchmarks and repo available).
AI-written summary · about
chapters (9)
- 0:00PyTorch scaling journey
- 1:44Inference evolution and components
- 3:10Low-latency inference with compile
- 4:17Training improvements with kernels
- 5:45Measuring throughput in training
- 7:32Scaling challenges and strategies
- 8:37Compile for distributed training
- 9:20Core readiness for production
- 9:40Call to action and repo
related talks
12mKeynote: AI Infrastructure
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
30mOur Journey With TorchTitan
24mTraining large scale models using PyTorch
13mFrom Silicon to Scale: Training at Scale
12mAI-powered infrastructure:
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads