Quantization impact on Apex performance
summary
Quantized Apex (Gemma 4 31B instruct) is evaluated on a DGX Spark across nine variants: 27 GPU hours, 4,344 runs, 6M input tokens, 891k outputs. BF16 is 61.4 GB at 3.3 tps; IQ4XS is ~16.7 GB, 11.3 tps, uses ~27% runtime memory and shows ~92% next-token agreement while Q80 hits ~97% agreement. 8-bit KV cache costs ~1% agreement, 4-bit ~6%. IQ2M breaks; a prompt template bug hides quality until fixed (57% → 86.5%), and a judge change raises measured precision (0.05 → 0.7).
AI-written summary · about
chapters (21)
- 0:00Introduction and the Apex case study
- 1:12Research objective
- 1:56What quantization is
- 2:24Quantization levels and naming
- 2:52Hardware and model baseline
- 4:25Evaluation setup and agentic failures
- 4:51Speed and model-size results
- 6:14Measuring quality
- 6:41Distribution-level comparison
- 7:41Next-token agreement
- 8:19Question-level behavior
- 10:17Nine model variants compared
- 10:50Agreement results
- 11:24The 4-bit tradeoff
- 13:47Agentic benchmark suite
- 14:38The practical sweet spot
- 15:06The low-bit performance cliff
- 16:56An unexpected result
- 17:27Rerunning the evaluation
- 18:17Summary
- 18:45A three-step selection method
related talks
19mGemma 4 Deep Dive
27mDeepdive into Gemma 3
09mOpen models and open compression
14mGemma on mobile and web. Best and worst practices
40mUnlocking Local LLMs with Quantization
36mPanel Discussion
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
10mDemo: Using Gemma with the Hugging Face ecosystem