CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
summary
Claims a combined algorithm–hardware co‑design that prunes and customizes LLMs to fit edge memory/energy budgets, implements a 28nm scalable accelerator with a specialized NPU and LPU for head swapping, and runs on two off‑the‑shelf edge platforms. Uses an encoder–decoder embedding plus grid search to produce layer‑wise pruning ratios, a per‑request adapter rebuilt from expert “mini‑tasks,” and a token‑level DFS tuner. Reports up to 11.92× speedup and 7.36× energy savings while preserving generation quality.
AI-written summary · about
chapters (6)
- 0:00Edge LLM deployment
- 3:32Edge device challenges
- 6:26Optimizing edge LLM inference
- 7:32CLONE architecture
- 12:18Performance benchmarks
- 16:00Contributions and future work
related talks
26mMemory-Efficient LLM Inference on Edge Devices With NNTrainer
24mCost and Performance Optimization of LLM Inferencing.
23mLLMs on the Edge: The Future of On-Device Intelligence
17mADAngel: Accelerating Arbitrary-Precision Quantized LLMs with Adaptive Computing Mapping
20mGAMING: Benchmarking Sustainable LLM Deployment on Edge Devices
08mHacks to Make LLM Training Faster
24mLLM Inference Optimization From Simple To Sophisticated
34mOptimizing AI Inference with ML Compilers & Hardware