MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on(…)
summary
Runtime called MemSpec implements a lightweight predictor, a draft-model cache manager and a runtime controller on a Jetson Orin Nano, running 7B target models (Llama 2, Qwen 2.5 quantized with GPTQ) with ~0.5B drafts. It finds non-resident loading costs 2.7× a verify iteration, cuts rollback execution below 5%, keeps prediction overhead <3.9%, and improves steady-state throughput by 40.7% over bandit MAB while reaching 95–97% of an oracle dynamic upper bound. prediction-guided resident working-set management and proactive memory-aware scheduling enable the gains.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:07Speculative decoding background
- 3:58Limitations of adaptive draft selection
- 6:24MemSpec design principles
- 7:50MemSpec architecture
- 9:20MemSpec runtime workflow
- 10:27Experimental setup
- 11:50Performance results
- 13:36Component contributions
- 14:42Summary
- 15:30Q&A
related talks
14mFreedom from memory interference
26mMemory-Efficient LLM Inference on Edge Devices With NNTrainer
15mA Managed Memory System for micro controllers with NOR flash memory
56mAutomatically Scalable Computation
27mStageML: Partial Evaluation for Multi-Tenant MoE-LoRA Inference
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
17mA Difference World: High-performance, NVM-invariant, Software-only Intermittent...
17mSparseZETA: Intelligent Auto-tuner for Designing High-Performance SpMV Programs