Sub-Linear Algorithms meets Large Language Models
summary
Argues that inference computes mostly zeros — measured 95% zeros in attention for a 175B model — and proposes dynamic per-input parameter selection using locality-sensitive hashing plus count sketches to sample active neurons. Demonstrates a production C++ system that trains a 7B model on CPUs, claims CPU inference can beat GPU baselines in some tasks, and replaces embedding-heavy RAG with learned pointer-generation; reports implemented system and empirical speed-memory wins.
AI-written summary · about
chapters (10)
- 0:00Sub-linear algorithms and LLM challenges
- 2:28LLM scale vs architecture
- 6:00Extreme sparsity in LLM computations
- 9:55Dynamic sparsity in neural networks
- 12:05Locality sensitive hashing for selection
- 16:23Adaptive dropout with LSH
- 23:25ThirdAI: theory to production
- 27:42Beyond embeddings for RAG
- 31:37Need for more algorithms
- 33:00Q&A: CPU vs GPU and scaling
related talks
16mInfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV...
30mLLM Inference Arithmetics: the Theory behind Model Serving
40mIs my Large Language Model a Strange Loop?
11mKeynote: Building a Large Model Inference Platform for Heterogeneous Chinese Chips Based on VLLM
24mCost and Performance Optimization of LLM Inferencing.
34mOptimizing AI Inference with ML Compilers & Hardware
01h 11mThe Move Toward AGI: Why Large Language Models Surprised Almost Everyone... | Theoretically Speaking
19mSystem for Scaling Large Language Models