NeuOS: A Latency-Predictable Multi-Dimensional Optimization Framework for DNN-drive
summary
NeuOS implements coordinated layer-level DVFS and low-rank approximation with precomputed hashed speedup-power tables to schedule multiple DNNs on Jetson TX2 and AGX Xavier. It evaluates AlexNet/ResNet/VGG across cohorts, finds about 68% energy improvement on TX2 (≈46% on Xavier) and a ~3% deadline-miss rate with 99.9th-percentile tail latency acceptable. The system requires one-time table/model generation (12k DVFS states on TX2, 52k on Xavier) and shows microsecond runtime overhead.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:17The tale of two worlds
- 1:31Marriage between the two worlds
- 2:01The big picture
- 2:45Where system software/framework can he
- 3:45Jack of all trades, master of none
- 9:08No one is alone
- 10:01Design Goals
- 11:32Timing predictability
- 12:51Coordination
- 15:45Optimization
- 17:12Overview
- 19:12Energy
- 21:34Tail Latency
- 22:03Relative Accuracy
- 22:44Flexibility
- 24:09Overhead
- 24:58The system community to the rescue
related talks
18mDeploying ML solutions with low latency in Python
37mDeep Learning Neural Network Acceleration at the Edge
14mLightning Talk: Every Millisecond Counts: The Fine-tuning Journey of an Ultra-Eff...
19mOptimizing CNN Model Inference on CPUs
22mMonoNN: Enabling a New Monolithic Optimization Space for Neural Network Inference...
38mAutoML for Efficient Neural Architecture Design
31mLow-Latency Solutions for Resource-Constrained Environments
23mThe Race to the Bottom: Low Latency in the age of the Transformer