On‑Device LLMs on Android Devices 2025
summary
An Android Compose app implements and compares three on-device LLM approaches: Google Gemini Nano via AICore (Pixel 9 only) which gives the best response quality and low power use but is device‑limited and can require reboots when it loops; llama.cpp (gguf via JNI) which runs quantized Llama models (e.g., 3.21B ~2.3GB or tinyLlama 1.1B ~640MB) offering model choice and deep customization at the expense of CPU battery use and higher latency; and MediaPipe LLM Inference (LiteRT → NNAPI/GPU) which runs 1B–3.7B models with about 4s summarization latency and good memory efficiency.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:36On-device LLM approaches
- 7:58Gemini Nano implementation
- 13:06Llama.cpp setup
- 18:03MediaPipe LLM API
- 22:47Benchmarking performance
- 30:37Practical use cases
- 34:18Summary
related talks
23mLLMs on the Edge: The Future of On-Device Intelligence
22mThe Disruptive Potential of On-Device LLMs
01h 21mTLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM
47mOwn Your LLM: Running Generative and Agentic AI Locally
21mFrom 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents
35mThe Android Lens: Applying Mobile Forensics to AI Performance
51mJlama: A Native Java LLM inference engine
11mRunning LLMs on your iPhone: 40 tok/s Gemma 4 with MLX