Tips & Tricks for Fast Neural Net Inference in Production
summary
Check speed early during model construction and training and prefer simple architectural moves that actually cut work: use 1x1 convolutions to cheaply reduce channel count, replace conv+pool with stride convs, and pick channel widths that are multiples of 8. Replace full convs with depthwise separable convs (MobileNet pattern) or apply low-rank/SVD and tensor decompositions to turn one heavy layer into two smaller ones, then fine‑tune. Apply structured pruning of whole filters/maps, quantize to FP16/INT8 with calibration, fuse BatchNorm into conv, and run with TensorRT and Tensor Cores for real 2x+ speedups.
AI-written summary · about
chapters (16)
- 0:00Inference introduction
- 1:31Key model metrics
- 3:16Training optimization
- 7:12Building from scratch
- 8:57Practical model tricks
- 15:36Advanced optimization
- 20:51Neural network pruning
- 25:49Matrix factorizations
- 26:10Optimal model training
- 29:21Model quantization
- 31:08Convolution acceleration
- 36:04Model graph optimization
- 39:40Hardware selection
- 42:51Acceleration libraries
- 45:40Summary
- 46:02Q&A
related talks
39mCreating smaller, faster, production-worthy mobile machine learning models for Android
43mNeural Networks: Fast GPU Inference with TensorRT
18mDeploying ML solutions with low latency in Python
16mLightning Talk: Efficient Inference at the Edge: Performance You Need at the Lowest...
08mHacks to Make LLM Training Faster
21mThings I learned running neural networks on microcontrollers
14mOptimize your TensorFlow Lite models
23mThe Race to the Bottom: Low Latency in the age of the Transformer