PetS: A Unified Framework for Parameter-Efficient Transformers Serving
summary
PetS is a serving framework that decouples PET methods into shared operators and lightweight task-specific PET operators, implements a Python front end with a C++ backend on a Turbo Transformers backend, and adds coordinated batching plus multi-cuda-stream operator scheduling. Evaluation on Jetson TX2, GTX 1080 Ti and V100 shows up to 25× more concurrent tasks and 1.53× (desktop) / 1.63× (server) throughput versus single-task baselines. It improves throughput but does not optimize latency and yields little gain on very constrained devices.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:27We Are in the Transformers Era!
- 0:49Pretrain-then-finetune Paradigm
- 1:31Explosion of Down-stream Tasks
- 2:01Parameter-Efficient Transformers (PETS)
- 2:47Challenges of PETS Serving
- 3:50Unified Representation of PETS
- 5:04Pets Overview
- 6:09User Interface
- 6:29Optimization Strategies
- 7:22Coordinated Batch Scheduling
- 10:08Evaluation Setup
- 11:06Maximum Number of Supported Tasks
- 13:04Serving Throughput with Fixed-Size Inputs
- 14:06Execution Time Breakdown
- 14:46Batch Scheduling Performance
- 15:40PET Operator Scheduling Performance
- 16:45Summary of Contributions
related talks
15mPET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated
12mPreview: ML Serving
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
16mGalvatron: An Automatic Distributed Training System for Efficient Large...
37mTraining and Optimisation of Large Transformer Models: An ATLAS and CERN Use Case - Ricardo Rocha, CERN & Maxence Draguet, University of Oxford
23mThe Race to the Bottom: Low Latency in the age of the Transformer
12mAI-powered infrastructure:
42mCloud TPU Pods: AI Supercomputing for Large Machine Learning Problems