Serving Pytorch Models in Production
summary
A BERT-based product-type classifier is already trained and then distilled and quantized to cut latency and size: F1 jumps from 0.75 to 0.91 versus the old model, distilled latency drops from ~112 ms CPU / ~25 ms GPU to ~83 ms with dynamic quantization, model file shrinks 267 MB → 138 MB, and end-to-end serving lands under single-digit ms with distillation+quantization+optimized runtime. Packaging uses the torch model archive with a custom handler, TorchServe exposes management APIs, server-side batching, and metrics so multiple model versions and A/B tests run in production.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:33Walmart search ML
- 3:37Production infra needs
- 5:45Dataset training overview
- 7:58Traditional to transformer
- 11:48BiLSTM vs BERT performance
- 12:31Hugging Face inference
- 16:05Latency CPU vs GPU
- 22:12Reducing inference latency
- 29:24Model distillation
- 31:11TorchScript conversion
- 34:06Quantization optimization
- 42:41Serving PyTorch models
- 46:10TorchServe packaging
- 50:05Deploying model server
- 54:41Production metrics
- 59:43Q&A
related talks
35mClient Side Deep Learning Optimization with PyTorch
10mQuantization
11mOptimizing Model Inference with PyTorch 2.0
14mOptimize your TensorFlow Lite models
21mEfficientBERT: Trading off Model Size and Performance
36mHigh-Performance Distributed Tensorflow
42mHigh Performance Distributed Tensorflow
14mFlexible Deployment of PyTorch Models on MCU-Class...