Rust for ML Infrastructure: A Case Study in Building an ONNX Runtime
Talkat RustConf 2026 · on Thursday, 2026-09-10· at 11:50· Room 510ac | Palais des Congrès de Montréal, Palais des Congrès de Montréal·
summary
A Rust ONNX runtime called Onyxia implements a small, 17-variant primitive IR and a composite lowering layer, built across eight crates and a WGPU backend that emits WGSL compiled by Naga. A solo engineer prototypes a working GPU inference engine in about 15 days of weekend work, trades peak perf for simplicity, and runs ~1,000 fast tests locally. Benchmarks on an RTX 5090 show Onyxia beating ONNX Runtime on single-token decode but losing on prefill; codebase stays tiny versus ORT’s ~2M C++ lines. WGPU + WGSL via Naga
AI-written summary · about
chapters (12)
- 0:00Introduction to talk
- 1:21What is ML infrastructure
- 3:17What is an ONNX model
- 8:27Why build a new runtime
- 10:36Rust’s role in the project
- 17:33Building with AI tools
- 18:19Rust toolchain benefits
- 20:23Type system for extensibility
- 23:00Backend implementation with Rust
- 24:09Testing and refactoring
- 25:39Rust ecosystem advantages
- 28:01Key lessons learned
related talks
01h 00mFrom Tiny Chips to Server Racks: Deploying AI Everywhere with Rust
25m"Rust for AI & Accelerated Computing"
05mONNX Runtime
30mScalable ML acceleration with ONNX Runtime
23mThe Race to the Bottom: Low Latency in the age of the Transformer
10mDeploying machine learning in wasm to CPU, GPU and NPU with the new wa...
19mEnhancing Gen-AI with Rust: Scalable AI Systems
30mThe Evolution of ONNX: Driving Interoperability in AI...