TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM
summary
Shows how LiteRT-LM packages a model as one deployable file and runs on Android and iOS with optional NPU AOT compilation, benchmarks thousands of tokens/sec on high‑end phones and about 133 tokens/sec on Raspberry Pi, and supports on‑device function calling and progressive disclosure of skills to save context tokens. Describes Gemma 4 E2B/E4B performance, a 270M Function Gemma achieving ~85–90% reliability on ten function calls, constrained decoding for tool safety, and a fine‑tuning workflow using synthetic data to ship sub‑1B production models.
AI-written summary · about
chapters (21)
- 0:00Intro: AI on the Edge, Small Language Models, and Gemma
- 4:51Enabling App Development: MediaPipe, LiteRT, and System Services
- 9:09Small Language Models: Performance, Reach, and Fine-tuning
- 11:30Gemma 4: Sizes (E2B and E4B) and AI Core Roadmap
- 16:10Gemma on Edge Runtime: Performance Benchmarks
- 18:34Agent Skills: Google AI Gallery, Mood Tracker, and Wikipedia Lookup
- 23:38Skill Architecture: Efficiency, Progressive Disclosure, and Tool Loading
- 27:34Reliability: Constrained Decoding and Tool Usage
- 29:18Community and Custom Skills
- 31:30Skill Development Deep Dive: Orchestrator and Registry
- 33:30Rapid Skill Prototyping: Using Gemini CLI and ADB
- 38:35Open Source: AI Edge Gallery and Community Engagement
- 41:00Deploying Tiny Models (sub-1B parameters) In-App
- 47:44Third-Party Models: Fast VLM and Hardware Acceleration
- 50:17Model Examples: Function Gemma, Mobile Actions, and Embedding Gemma
- 55:41AI Edge Eloquent: Transcription and Text Polishing
- 59:07Modularity Playbook: ASR and Text Polishing Engines
- 1:01:23Synthetic Data Workflows for Tiny Models
- 1:06:36Web Support and Fine-tuning Documentation
- 1:08:20Summary and Key Takeaways
- 1:12:49Q&A: Multi-skill Execution, Context Windows, and Future Roadmap
related talks
08mDemo: Gemma on-device with MediaPipe and TensorFlow Lite
14mGemma on mobile and web. Best and worst practices
35mOn‑Device LLMs on Android Devices 2025
23mLLMs on the Edge: The Future of On-Device Intelligence
39mRunning On-Device AI With Qualcomm AI Engine Using LiteRT & ExecuTorch
47mOwn Your LLM: Running Generative and Agentic AI Locally
12mDemo: Optimizing Gemma inference on NVIDIA GPUs with TensorRT-LLM
31mFrom ML to LLM on device AI in the browser