Optimizing CNN Model Inference on CPUs
Talkat USENIX ATC '19 · on Friday, 2019-07-12· at 11:50· USENIX ATC Track II: Grand Ballroom VII–IX·
summary
NeoCPU replaces third-party kernels with tunable operator templates (block sizes, register count, loop unrolling) and records real-device timings per configuration, then runs a dynamic-programming graph tuner to pick inter-layer data layouts and when to insert transforms, trading local optimality for global latency. Results show up to 3.45× lower latency versus MXNet/OpenVINO baselines, consistent wins on Intel, AMD and ARM CPUs, uses CHW16c layout and transformed kernels, and adds a custom lock-free thread pool for better multi-core scaling.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:55CNN model inference on CPUS
- 1:43Deep Learning Frameworks
- 2:46Deep Learning Compiler
- 4:36Optimize convolution on CPUs
- 7:26Data Layout Coordination
- 12:56Evaluation - Performance
- 14:15Evaluation - Scalability
- 15:27Conclusion
related talks
22mMonoNN: Enabling a New Monolithic Optimization Space for Neural Network Inference...
23mThe Race to the Bottom: Low Latency in the age of the Transformer
29mEfficient AI with Python: running AI models on CPUs
23mOptimize your network inference time with OpenVINO
26mNeuOS: A Latency-Predictable Multi-Dimensional Optimization Framework for DNN-drive
14mKairox: Adaptive GPU-CPU Hybrid LLM Inference via Online Neuron Balancing
25mCMSIS-NN
24mTools & Techniques for a cost optimized Analytics & AI pipeline