Scaling Inference of O(10K)-length Sequence Recommendation Models Using...
summary
Pinterest scales sequence length from 500 to 16,000 for a 10M+ inferences/sec ranking fleet while keeping P90 <50 ms by combining TorchScript + a custom CudaGraph backend and Triton kernels, storing all transformer weights in GPU L2 with a Single Kernel Unified Transformer that cuts latency 85% and memory 13% vs FlashAttention for small layers. System fixes include request-level deduping and a pinned memory arena, plus a custom sparse tensor format that yields 16x payload shrinkage, removes PCIe saturation, and collectively drives ~250x P99 E2E latency reduction and ~98% lower GPU serving cost.
AI-written summary · about
chapters (13)
- 0:00Scaling sequence models
- 3:53Sequence processing pipeline
- 6:21System bottlenecks
- 8:06Request duplication optimizations
- 10:12Pinned memory arena
- 12:14Model-level optimizations
- 12:36Custom Triton kernels
- 14:47Lean transformer optimizations
- 17:20Sparse attention top-k
- 18:24Wide transformer fusion
- 19:47Latency and throughput gains
- 20:29Optimization takeaways
- 21:32Q&A on bottlenecks
related talks
10mLightning Talk: Scaling Recommendation Systems To 2K GPUs and Beyond
26mEfficiently serving LLMs at scale.
23mThe Race to the Bottom: Low Latency in the age of the Transformer
13mFrom Silicon to Scale: Training at Scale
30mLLM Inference Arithmetics: the Theory behind Model Serving
32mScaling Inference Deployments with NVIDIA Triton Inference Server and Ray Serve
21mLessons Learned in WatsonX Training: Scaling Cloud-Native...
10mKeynote: How to Leverage PyTorch to Scale AI Training and Inferencing