Libra: Flexible Request Partitioning and Scheduling for Serving Unbalanced and Dynamic...
summary
Libra implements a micro-request flexible partitioning that splits requests at any token boundary, a global scheduler that binary-searches a split point (≤6 iterations) to equalize per-GPU service rates, and per-GPU local batching that tunes prefill:decode ratios and decode context. It ships chunked KV cache transfers with ~94% overlap to hide transfer cost, runs on A100/H100 setups, and on real traces raises goodput up to 1.91× vs colocated and 1.61× vs disaggregated, increases capacity up to 3.07×, and cuts latency tails substantially.
AI-written summary · about
chapters (11)
- 0:00Libra introduction
- 1:41LLM inference challenges
- 3:05Dynamic imbalance in workloads
- 4:30Static partitioning limits
- 6:17Libra’s flexible paradigm
- 8:50Global scheduling for splitting
- 11:40Local batching under SLO
- 13:48KV cache transfer
- 14:30Experimental results
- 15:34Libra in production
- 16:19Conclusion
related talks
20mScheduler and Sharding Considerations for Network Efficiency
19mEfficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance...
33mLibraBFTv2: Optimistically-linear BFT Consensus with Concrete Latency Bounds
15mDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
16mTowards Optimal Rack-scale μs-level CPU Scheduling through In-Network Workload...
32mDynamic Scheduling for Large Language Model Serving