Aggressive LLMs Optimization: Making Them Work on Tiny Devices
summary
Uses GPT-2 117M / 500MB as a baseline, fine-tunes it to generate ISO software requirements, then applies 12→6 layers and 768→384 hidden architecture trimming, magnitude-based pruning, and int8 quantization. Distillation fails after aggressive trimming. Result: model size shrinks roughly 5×, CPU load drops from about 45% to near 12% with combined methods (at cost of ~+200MB RAM), perplexity degrades only slightly and outputs remain usable for the focused task.
AI-written summary · about
chapters (7)
- 0:00Introduction and Speaker Introduction
- 0:31The Challenge of Optimizing Large Language Models
- 1:47Choosing the Right Model: GPT-2
- 3:48Optimization Techniques: Theory
- 9:56Practical Research and Experiments
- 18:42Combining Optimization Methods
- 21:14Conclusions and Final Takeaways
related talks
20mLLM: Efficiency and Sustainability in Practice
27mProductionizing Open Source LLMs
11mWhen Less Is More: A Technical Overview of LLMs and the Strength of Smaller Models
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
29mRunning Large Language Models on Laptops...
25mFinetuning Large Language Models (LLMs) for Security Log Detections
19mSystem for Scaling Large Language Models
22mThe Disruptive Potential of On-Device LLMs