Frontier results, on device
summary
A concrete pipeline uses Phoenix (Arize) to run capability and production-trace evals comparing Claude Sonnet, Gemma 4 E2B, Qwen 2.5 and Llama 3.2 3B against a curated golden dataset for a conversation-summarization feature called Mima. Results show Llama 3.2 3B with a few-shot prompt and lightweight postprocessing matches Claude’s accuracy while cutting cloud cost, yielding saved ~$1/day, P50 ~1s P95 <3.5s, 100% JSON validity and ~93% factual consistency.
AI-written summary · about
chapters (15)
- 0:00The hidden costs of AI
- 1:10Analyzing inference expenses
- 2:51Introduction to SLMs
- 4:33Evaluating local model performance
- 6:52Local AI in practice
- 8:47Framework for right-sizing AI
- 10:47Setting up golden datasets
- 12:58Model selection process
- 15:55Finding the 'Sage' model
- 19:22Optimizing with prompt engineering
- 21:12Iterative prompt testing
- 23:04Closing the performance gap
- 25:40Post-processing for reliability
- 27:45Regression and maintenance
- 28:36Next steps for local adoption
related talks
31mFrontier results, on device
55mBuild enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform
52mHW, SW, Performance and Costs for Llama-2 70b and Mixtral 8x7b LLM Inference with Low...
25mFrontier AI Models: A Review of Past Nine Months
48mBringing Llama 3 to Life
36mBuilding & Deploying LLM Apps
22mThe Disruptive Potential of On-Device LLMs
Stop Paying for Frontier Models