StageML: Partial Evaluation for Multi-Tenant MoE-LoRA Inference
summary
StageML applies partial evaluation to multi-tenant MoE+LoRA serving, using a binding-time lattice to decide what can be precomputed at deployment or adapter load and what stays token-dependent. It residualizes adapter computations and uses a memory-aware planner to materialize hot adapters under VRAM budgets. On Mixtral 8×7B with 2GB budget StageML removes 52.34% of adapter computation from token time and yields a 1.105× end-to-end speedup; when fed to a fused-experts backend expert-phase runtime shows up to 2.806× speedup.
AI-written summary · about
chapters (8)
- 0:00StageML for LLM inference
- 1:47LLM inference and LoRA adapters
- 6:23Multi-tenant MoE systems
- 11:14Partial evaluation optimization
- 16:40Runtime correctness
- 18:25Experimental setup and results
- 22:08Contributions and limitations
- 23:31Q&A
related talks
12mVertical Mobility: Inference from MVP to Trillion Parameters
20mAccelerating Distributed MoE Training and Inference
24mLLM Inference Optimization From Simple To Sophisticated
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
15mResource Multiplexing in Tuning and Serving Large Language Models
25mOptimizing Reinforcement Learning at Trillion-Parameter Scale
14mToppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference
12mSmartMoE: Efficiently Training Sparsely-Activated Models through Combiningand...