Auto-tuning vllm
summary
Autotuning vLLM experiments use Optuna (TPE, BoTorch, grid) and MLPerf benchmarks to search config space for Qwen3-32B-FP8 and Qwen3-30B-A3B-FP8, comparing against vLLM defaults. Results show a best-case 35% throughput increase while achieving an 8.5% latency decrease, earlier runs give ~30% throughput on Qwen and ~9% tokens/sec gain on Llama 3.1 on a single L40S. Harness runs parallel trials across GPUs and proves autotuning viable.
AI-written summary · about
related talks
26mVLLM-Helion: SOTA LLM Performance by Advanced Autotuning and Fine-grained Dispatching
31mHow IBM Research Achieved vLLM Platform Portability with Triton Autotuning
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
09mLightning Talk: Vllm-triton-backend: How To Get State-of-the-art Performance on...
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
29mLocal LLM fine-tuning: a practical example
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
03mFinetune Llama 4 Faster