Run Frontier AI at Home
summary
Claim: a 100x gap in cost or speed sits in the stack, not the model. Running GLM 5.1 (1T) across four Mac Studios currently costs ~$40k and hits ~20 tokens/sec; real gains come from concrete changes: kernel fusion recovered 30% on Qwen 3.5, RDMA cuts node-to-node latency from ~300µs to single digits, and offloading prefill to an RTX Spark halves large‑prompt inference. The demo shows multi‑node Thunderbolt clusters, heterogeneous splitting of prefill/decode, and measurable token‑rate improvements.
AI-written summary · about
chapters (20)
- 0:00Introduction to EXO Labs and the mission to democratize frontier AI
- 2:00The current state of AI: centralized cloud systems vs. the need for local infrastructure
- 7:40Technical challenges: kernel efficiency and the overhead of separate kernel launches
- 9:50The importance of the software harness in optimizing inference performance
- 10:35Understanding inference constraints: compute-bound vs. memory-bound operations
- 11:28The distinction between prefill and decode phases in LLM inference
- 13:07Requirements for efficient local decoding: memory capacity, bandwidth, and energy efficiency
- 15:50The concept of 'Intelligence per Joule' as a performance metric
- 16:45Advancements in consumer hardware: higher memory capacity and bandwidth on Apple Silicon
- 18:50Q&A: The future of consumer appetite for local inference hardware
- 20:17Discussing the cost and performance of running trillion-parameter models like GLM 5.1
- 22:05The 100x potential: How code design across the stack improves performance
- 26:35Future outlook: Bifurcation of local vs. cloud use cases and diminishing returns of model size
- 38:55Heterogeneous hardware strategies: Combining compute-dense and bandwidth-heavy devices
- 41:47Demo: Using an Nvidia RTX Spark to accelerate prefill on a Mac cluster
- 48:40Software architecture: Automating cluster orchestration with EXO
- 53:00Challenging the 'batching' necessity: Multi-agent systems, search, and continual learning
- 1:05:22Rethinking cloud economics and renting use cases instead of hardware
- 1:20:55Demo technicalities: Event sourcing and cluster node discovery
- 1:32:20Closing thoughts: Transparency in benchmarks and the future of open-source model evaluation
related talks
28mServing a Frontier Open Model with Ray & vLLM
35mLocal LLMs on low end hardware - a practical perspective
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
30mRay Across the Stack Fine Tuning Qwen and Serving 3D Generation at Autodesk with AWS Trainium and In
20mFireworks AI: Go from Hype to High - Value AI
30mGo, ONNX and AI in Production
01h 18mOptimizing LLM Training and Inference Performance on GPUs
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality