Running an Open Source LLM on Open Source Hardware
summary
A UC Santa Cruz project builds an 1.58-bit accelerator that compiles a full transformer inference pass to an instruction stream and runs on custom functional units for ternary matrix multiply, RMS norm, and rowwise arithmetic. Models (370M, 1.3B, 2.7B) use weights in {-1,0,1} so multiply–accumulate reduces to adds, shrinking storage; a software emulator validates correctness before synthesis. The design scales from 30 tokens/second on Artix‑7 to >1000 tokens/second on Alveo U250, demos on a NiteFury II PCIe board, and an open‑toolchain workflow favors sv2v preferred over slang/yosys-sv with Verilator issues tracked.
AI-written summary · about
chapters (12)
- 0:00Turnip project intro
- 1:30Matmul-free LLM algorithm
- 2:54Turnip accelerator units
- 3:59FPGA performance
- 5:04Open source tooling
- 7:31Live demo and challenges
- 9:59Project journey and team
- 10:42LLMs for open source dev
- 13:32LLMs in hardware future
- 15:16Open source in agentic era
- 16:49Q&A: Architecture & power
- 21:48Q&A: LLM sustainability
related talks
17mLLM Inference on RISC-V Embedded CPUs
25mSPARQ: An Accelerator Architecture for Large Language Models with Joint Sparsity and(…)
24mCelerity: An Open Source 511 Core RISC V Tiered Accelerator Fabric
11mDemo: Running Transformers on Semidynamic's "All-In-One" Vector and Tensor Unit
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
34mOptimizing AI Inference with ML Compilers & Hardware
22mAdvancing AI Computing With Optimized Matrix Multiplication Techniques With RISC-V CPU
29mLeveraging open-source LLMs for production