The future of Edge Agents
summary
Presents running Mistral 3B (3 billion parameters) efficiently on mobile/edge devices by mixed quantization to int8/int5 to fit in a few GB of RAM while matching 7B benchmark accuracy and sustaining ~10 tokens/sec. Explains solutions for KV-cache bottlenecks and using interleaved sliding windows for longer context, and demonstrates native function-calling plus an agent pipeline that emits tool calls, executes them, and returns structured JSON results end-to-end.
AI-written summary · about
chapters (10)
- 0:00Edge Agents intro
- 2:22Edge devices challenges
- 4:12Mistral 3B model
- 7:04Optimizing with quantization
- 10:20Speed vs accuracy
- 12:50Near-reference Mistral 3B
- 14:40Sliding window attention
- 15:00Agents and function calls
- 17:36Building agents with tools
- 19:12Future of Edge Agents
related talks
20mPyTorch Edge: Developer Journey for Deploying AI Models Onto Edge Devices
29mEnhancing AI with Edge Computing: A Leap Towards Affordable Performance
21mUnlocking the potential of low-power embedded devices with Edge AI software
24mAccelerating AI on Edge
12mMistify: Automating DNN Model Porting for On-Device Inference at the Edge
27mInferencing at the edge and Fragmentation Challenges
48mThe Power of Edge AI: Revolutionizing Future of Real-Time, Autonomous Processing
25mLeveraging PyTorch for Generative AI in Distributed Edge Clouds