From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents
summary
Function Gemma ships at 270 million parameters and runs almost 2,000 tokens per second prefill on a Pixel 7, initially scoring 46% on app intents and, after fine‑tuning on synthetic data, reaching >90% on eight of ten functions. A live restaurant roulette skill runs fully on-device and Eloquent chains two sub‑hundred‑million models for robust transcription. The stack uses system LLMs like Gemma 4 via AI Core and app‑embedded models with LiteRT-LM, with an export flow from PyTorch to on‑device runtime.
AI-written summary · about
chapters (18)
- 0:00Introduction to on-device agents and tiny LLMs
- 0:48Overview of AI Edge, SLMs, and TLMs
- 1:06Taking a look at tiny models
- 1:24Motivations for on-device AI (latency, privacy, offline use)
- 3:01System-level GenAI (Gemini Nano via AI Core)
- 4:03App-level GenAI (LiteRT-LM for custom/boutique models)
- 5:06Google AI Edge Gallery app demo
- 6:22Deep dive into agent skills and the skill harness
- 7:41How the skill harness works (system prompts, tool calls, and JavaScript UI)
- 9:00Creating and publishing your own skills
- 10:28Using LiteRT-LM runtime for model deployment
- 12:31Export and inference workflow (from PyTorch to deployment)
- 13:19Function Gemma: Robust, small-scale function calling
- 14:35Fine-tuning workflow for tiny models using synthetic data
- 16:01Eloquent: A production transcription app example using tiny models
- 17:28Q&A: Agent skill robustness and multi-skill calling
- 19:26Q&A: LiteRT-LM file format vs. Task files
- 20:00Q&A: Performance on CPU/TPU and resources
related talks
08mDemo: Gemma on-device with MediaPipe and TensorFlow Lite
46mSmall AI is the next Big Thing
48mBuilding Offline AI Agents in Your Mobile App
05mBuild your own LLM on Cloud
23m2025: On‑Device AI Goes Mainstream on Android
35mOn‑Device LLMs on Android Devices 2025
41mBuild intelligent Android apps with Google's AI
47mOwn Your LLM: Running Generative and Agentic AI Locally