Advancements in optimizing ML Inference at CERN
summary
SOFIE converts Keras/PyTorch/ONNX models into standalone C++ plus weight files with only a BLAS dependency, claiming support for ~75–80 operators and idiomatic models like ParticleNet and ATLAS GN2. Ahead-of-time layer fusion plus greedy interval scheduling memory reuse and constant folding cuts peak memory dramatically; benchmarks show much lower memory for ParticleNet (~500,000 parameters) and ATLAS GN2 (~1M) versus ONNX Runtime, similar latency for many cases, prototype runs on NVIDIA via Alpaka and exposes a unified BLAS interface called SophieBLAS while still working on tighter latency for large models.
AI-written summary · about
chapters (11)
- 0:00ML at CERN challenges
- 3:21Triggers in data filtering
- 5:06ML inference pipeline challenges
- 7:10Current ML inference tools
- 9:20Sophie project for inference
- 12:33Sophie layer fusion optimization
- 15:43Sophie inference efficiency
- 18:10Sophie vs ONNX runtime
- 19:14Heterogeneous inference with Sophie
- 23:05Future directions at CERN
- 25:33Conclusion
related talks
12mPreview: ML Serving
18mDeploying ML solutions with low latency in Python
43mLightweight, low-overhead, high-performance: machine learning directly in C++
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
30mScalable ML acceleration with ONNX Runtime
19mThe Small Model Infrastructure Nobody Built (So We Did)
30mThe Evolution of ONNX: Driving Interoperability in AI...
36mAccelerating High-Performance Machine Learning at Scale i...