LitePred: Transferable and Scalable Latency Prediction for Hardware-Aware Neural...
Talkat 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24) · 2024-04-16..04-18·
summary
LitePred claims it predicts DNN inference latency on new edge platforms by transferring existing predictors rather than rebuilding them. It uses a VAE data sampler to generate in-distribution NAS configurations and a latency distribution-based similarity detection to pick which pretrained predictor to fine-tune. Evaluated on 85 edge platforms and 6 NAS search spaces, it reports 99.3% average prediction accuracy, less than an hour of adaptation, up to 5.3% higher accuracy and a 50.6× reduction in profiling cost.
AI-written summary · about
chapters (11)
- 0:00LitePred introduction
- 1:01Hardware-aware design challenges
- 2:48Existing latency prediction limits
- 3:51Data distribution alignment challenges
- 4:53Efficient data sampler
- 6:17Latency predictor architecture
- 7:21Transferable latency prediction insight
- 8:45Platform adaptation similarity
- 9:29Experimental results comparison
- 10:16Similarity detection effectiveness
- 10:38Conclusion & Q&A
related talks
37mDeep Learning Neural Network Acceleration at the Edge
42mML Latency No More- Useful Patterns to Reduce ML Prediction Latency to Sub X ms
13mLeveraging AI for Faster Storage Access: A Graph-Neural-Network-Based Prefetcher
31mLow-Latency Solutions for Resource-Constrained Environments
06m⚡ Eliminating Adaptation Latency: Day-0 Inference for New Models on Multi-Backends With...
17mContent Aware Encoding for low latency live streaming encoders using deep learning
36mApplied AI on the edge
26mMachine Learning in Fastly's Compute@Edge