ADAngel: Accelerating Arbitrary-Precision Quantized LLMs with Adaptive Computing Mapping
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
ADAngel delivers a workload-adaptive mpGEMM engine: it implements a unified DPR model to generate padding, bitwise and a new split strategy (which cuts intermediate partial products 16× for W4A8), builds a Computation Strategy Set via Cutlass/PMA and FasterTransformer, and compiles an Oracle Policy Map (256 KB) from ~50.7 hours of offline profiling. On Jetson AGX Orin and A100 it achieves up to 5.10× decode throughput vs. llama.cpp and 1.17–2.38× prefill TTFT vs. TensorRT-LLM, validating the approach.
AI-written summary · about
chapters (14)
- 0:00ADAngel project intro
- 1:24Arbitrary precision quantization challenges
- 3:07Computation strategies for APQ
- 4:55Static kernels limitations in APQ
- 5:58ADAngel architecture workflow
- 7:03DPR unified computation model
- 8:31Padding bitwise and split strategies
- 10:39Offline profiling policy mapping
- 12:02Evaluation setup platforms
- 13:29Performance results spectrum
- 14:33Cross-platform A100 validation
- 15:16Ablation study design choices
- 15:56Key insights conclusions
- 17:05Q&A
related talks
23mLLMs on the Edge: The Future of On-Device Intelligence
17mCLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
12mDemo: Optimizing Gemma inference on NVIDIA GPUs with TensorRT-LLM
21mScaling LLM Inference with vLLM
08mHacks to Make LLM Training Faster
14mToppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…