Lightning Talk: Vllm-triton-backend: How To Get State-of-the-art Performance on...
summary
vLLM gains a single Triton-only code base that runs unchanged on NVIDIA H100 and AMD MI250, using 3D launch tiles and parallel softmax plus persistent-launch patterns to cut Triton JIT overhead. Autotuning and heuristic selection replace hand-written CUDA/HIP kernels; initial prototype is 6× slower than CUDA but optimizations close the gap and deliver a ~0.7% win vs FlashAttention3 on Hopper in Llama-38B decode benchmarks. Source, microbenchmarks and PRs are public; results hold across GPUs. autotuning plus CUDA-graph compatibility
AI-written summary · about
related talks
31mHow IBM Research Achieved vLLM Platform Portability with Triton Autotuning
26mVLLM-Helion: SOTA LLM Performance by Advanced Autotuning and Fine-grained Dispatching
10mAuto-tuning vllm
08mGPU Programming with Triton Kernels
12mDemo: Optimizing Gemma inference on NVIDIA GPUs with TensorRT-LLM
52mHW, SW, Performance and Costs for Llama-2 70b and Mixtral 8x7b LLM Inference with Low...
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
01h 40mFrom model weights to API endpoint with TensorRT LLM