QFactory: Accelerating Quantized Large Language Model Serving with Qtile Graphs
summary
QFactory is a compilation framework that generates fused quantized kernels by deferring dequantization, representing quantized tensors with a Qtile-graph abstraction, and applying graph-level transformations plus operator-level scheduling to pick memory-loading paths. It integrates into VLM-style serving, produces code for three GPU platforms, and claims an average 1.66× kernel speedup and 1.23× end-to-end gain versus prior compilers; evaluations also report up to 1.5× decoding speedup and outperforming a handcrafted library by ~1.3×. QFactory uses deferred dequantization to expand optimization space.
AI-written summary · about
chapters (11)
- 0:00QFactory introduction
- 0:40Serving quantized LLMs challenges
- 2:29Quantized operator limits
- 3:30Fine-grained quantization issues
- 4:55QFactory’s key insights
- 6:20Memory reuse in quantized ops
- 7:44QAI tensor abstraction
- 9:09Graph-level QAI propagation
- 10:34GPU memory scheduling
- 11:38Performance evaluation results
- 12:41Conclusion and future
related talks
17mQVM: Quantum Gate Virtualization Machine
38mOptimizing vLLM Performance through Quantization
21mScaling LLM Inference with vLLM
19mQuantum Virtual Machines
29mRunning Large Language Models on Laptops...
40mUnlocking Local LLMs with Quantization
45mRunning Your Coding Agent Locally: Lessons from a Real-World Experiment
16mInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...