On-Device LLM Inference on Android With ExecuTorch and Qualcomm QNN
summary
Shows how to run a full ViT‑B/32 CLIP pipeline ExecuTorch + Qualcomm QNN on Android, with model export via torch.export, AOT graph transforms, and on‑device delegates (XNNPACK, Vulkan, QNN). Demo runs live image/text cosine similarity on‑phone (they treat >20% as a match), uses post‑training quantization including W8A8 quantization, and requires adding Qualcomm AI Engine Direct JNI libs for NPU acceleration. Includes real latency, memory, and power tradeoffs and a working app.
AI-written summary · about
chapters (11)
- 0:00On-device AI with ExecuTorch
- 1:01Why on-device inference matters
- 3:47ExecuTorch and workflow
- 8:03PyTorch to Android deployment
- 11:37Quantization for edge devices
- 14:08ExecuTorch partitioning
- 15:52JNI and native AI stack
- 18:23Live CLIP demo
- 21:18CLIP inference code
- 22:00Android ML best practices
- 24:29Future use cases
related talks
37mHow to Optimize, Validate and Deploy ML Models On Device
26mHow to use Qualcomm AI Hub
39mMachine Learning on Android Demystified
46mFire Up Your Smartphone Neurons with On-Device Deep Learning
38mNEURAL NETWORKS ON YOUR MOBILE PHONE. DIVING INTO THE NN API
16mLightning Talk: Efficient Inference at the Edge: Performance You Need at the Lowest...
39mAI/ML on mobile/android with TensorFlow
01h 01mHow to Optimize, Validate and Deploy ML Models on Device