Optimizing vLLM for Intel CPUs and XPUs
summary
vLLM is integrated with Intel’s IPEX backend to run on Xeon CPUs and Intel GPUs/XPUs with zero code change for users; integration is op-level (KV cache, continuous batching, attention kernels) and pending mainline PR. Measured wins include AMX 15.11x vs AVX‑512 on matrix kernels, Xeon 6 showing ~1.9x gen‑over‑gen throughput and up to 5.5x AI inference TCO vs competitors, Llama 3 8B sees ~2x throughput on Granite Rapid, and W8A8 ~30% throughput gain over BF16.
AI-written summary · about
chapters (6)
- 0:00Intel hardware introduction
- 2:37SYCL and oneAPI ecosystem
- 9:04IPEX and PyTorch integration
- 13:40vLLM integration strategy
- 16:14Performance and benchmarking
- 28:42Q&A session
related talks
24mOptimizing CPU LLM Inference in PyTorch: Lessons From VLLM
22mEnabling VLLM V1 on AMD GPUs With Triton
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
44mIntel Math Kernel Library Perspectives & Latest Advances
11mGadi Singer (Intel) interviewed at the O'Reilly Artificial Intelligence Conference NY 2019
39mAccelerating Open-Source RL and Agentic Inference with vLLM
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees