Hardware acceleration for on-device Machine Learning
summary
Hardware acceleration reduces on-device inference latency: examples show GPU delegate running BlazePose about 184% faster than CPU and MobileNet seeing up to 10x improvement. Options include TensorFlow Lite via Google Play services, ML Kit, NNAPI/DSP/NPUs and the Model Analyzer to check GPU compatibility. Avoid GPU-unfriendly ops like BATCH_TO_SPACE and prefer RGBA 4-channel inputs; unsupported custom ops fall back to CPU and cause slow cross-device handoffs. An Acceleration Service in early access will benchmark device targets and recommend the best delegate.
AI-written summary · about
chapters (7)
- 0:00Introduction to ML on Android
- 1:07Mobile hardware landscape
- 4:43Implementation and benefits
- 7:47Adding GPU acceleration
- 10:00Optimizing models for hardware
- 12:22Managing hardware delegates
- 13:59Future tools and conclusion
related talks
21mHow to use hardware acceleration for Machine Learning inference on Android
13mWhat's new in Android machine learning
23mHugging Face Accelerate: Making Device-Agnostic ML Training and Inference Easy...
37mDeep Learning Neural Network Acceleration at the Edge
10mMachine Learning for mobile
26mIntroduction to Hardware Acceleration (en)
39mAI/ML on mobile/android with TensorFlow
31mBringing AI on-Device: From Cloud to Edge