Running Large Language Models on Laptops...
summary
Claims that recent Python tooling makes LLMs runnable on consumer laptops by using concrete quantization choices and measurements: compares QLoRA, bitsandbytes, GGUF/GGML and quantization-aware methods like AWQ/GPTQ; evaluates memory use on a 12 GB GPU, latency, and output quality across 15–20 miniaturized models trained on mental‑health corpora; finds an empirical sweet spot at 8‑bit (LLM‑int8) versus fragile 4/2/1‑bit results; demonstrates working local runs for multi‑billion models (examples include 27B and 70B families) with context‑size tradeoffs (8k vs 256k).
AI-written summary · about
chapters (6)
- 0:00Welcome
- 3:23Running LLMs on laptops
- 10:32Quantization techniques
- 17:41Use cases
- 19:07Q&A on quantization
- 29:00Closing
related talks
40mUnlocking Local LLMs with Quantization
15mQuantization at the Edge: Making a 4GB Model Run on 1GB RAM
47mOwn Your LLM: Running Generative and Agentic AI Locally
20mSelf-hosting AI LLMs - a beginners guide
27mProductionizing Open Source LLMs
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
19mQuant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric...
01h 02mA hacker's guide to open source LLMs