Running On-Device AI With Qualcomm AI Engine Using LiteRT & ExecuTorch
summary
Claims ability to run full PyTorch and TFLite models completely on-device using ExecuTorch + Qualcomm QNN (via LiteRT), with an AOT compile → partition → package (.pte) → JNI runtime pipeline. Recommends quantization like weight8/activation16, warns unsupported ops on NPU can crash execution, and emphasizes zero‑copy, warm‑up, never block UI patterns. Demo runs multimodal Gemma 4 on a Samsung S25 NPU at about 35–40 tokens/sec and shows the lightweight runtime and Maven-distributed .so libs actually work end to end.
AI-written summary · about
chapters (10)
- 0:00On-device AI & Qualcomm role
- 3:50Why on-device AI matters
- 7:47Quantization for edge models
- 10:56Model partitioning for CPU/GPU/NPU
- 14:26JNI layer for Android AI
- 15:55LiteRT for on-device inference
- 19:48Live demo: multimodal AI
- 24:37Deploying AI models on Android
- 27:44Future of agentic AI locally
- 29:58Q&A
related talks
01h 21mTLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM
24mAccelerating AI on Edge
12mDemo: Optimizing Gemma inference on NVIDIA GPUs with TensorRT-LLM
17mRunning Intelligence at the Edge: Web AI in Action
01h 19mWORKSHOP 2: AI FOR EVERYONE
21mFrom 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents
27mAccelerating Generative AI
01h 20mWorkshop1: AI for Everyone