BatchGen: An Architecture for Scalable and Efficient Batch Inference
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
An execution architecture represents every input sequence as a fine-grained event-driven coroutine and exposes sequence coroutine compute model primitives to pause, regroup, repartition, and move sequences across devices. Runtime keeps KV caches and weights on host, streams KV asynchronously, and uses offline profiling plus a brute-force search to pick yielding and offloading hyperparameters. Implementation, BatchGen, sustains utilization across MOE and long-tail generations, increases batch throughput (128‑GPU runs cut completion time up to 2.3×) and beats offloading baselines on memory‑constrained GPUs up to 9.6×; key APIs are yield, combine, partition, migration.
AI-written summary · about
chapters (14)
- 0:00BatchGen introduction
- 1:02Batch inference goals
- 2:05Why batch inference matters
- 4:09Batch inference challenges
- 5:13GPU underutilization issues
- 7:18Hardware diversity challenges
- 8:24Scheduling opportunities
- 9:50BatchGen architecture
- 13:05Profiling and policies
- 14:30Dynamic scheduling policies
- 15:36Performance gains
- 17:26Memory constraints
- 18:10Flexibility of BatchGen
- 18:32Summary and release
related talks
26mEfficiently serving LLMs at scale.
18mDVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient...
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
18mBatchy: Batch scheduling Data Flow Graphs with Service level Objectives
32mAnyscale's Ray Data: Revolutionizing Batch Inference
36mAccelerating High-Performance Machine Learning at Scale i...
33mHow Genmo Scales Video Generation with
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources