DVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient...
summary
DVABatch is a GPU runtime that slices DNNs into staged executors and exposes three meta-operations new/stress/split to implement a multi-entry multi-exit batching scheme; executors coordinate via batch queues and a state‑transition diagram to safely legitimise buffer pairs. It integrates with existing serving frameworks (Triton/tensorflow serving), targets input/operator/load diversity, and in evaluation reduces average latency ~46.4% and increases peak throughput up to 2.12×, showing the approach works.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:51Deploying DNN Services On GPUs
- 1:26Batching Improves Efficiency
- 2:45Inefficiency due to Diversities
- 6:12Weakness of Single-entry Single-exit
- 7:29Overview of DVABatch
- 8:29Holistic Multi-entry Multi-exit Scheme
- 9:19Management of stage executors
- 13:05Interface for Creating Policies
- 13:39Policies for Three Diversities
- 14:27Experiment Setup
- 15:00End to End Results
- 15:29Robustness at Stepping Load
- 15:58Contributions
- 16:36Implications
related talks
20mBatchGen: An Architecture for Scalable and Efficient Batch Inference
26mEfficiently serving LLMs at scale.
15mEffectively Scheduling Computational Graphs of Deep Neural Networks toward Their...
18mDISTMM: Accelerating Distributed Multimodal Model Training
19mML training acceleration with heterogeneous resources in Bytedance
18mBatchy: Batch scheduling Data Flow Graphs with Service level Objectives
03mDBTB INT Subbu Rama r
SpaceExit: Enabling Efficient Adaptive Computing in Space with Early Exits