Building Reliable AI Assistants with Small, Self-Hosted Language Models
summary
Built a production assistant that runs entirely on self-hosted models using a decomposed pipeline: four-layer architecture (action classification, tool selection, argument generation, answer synthesis) with a fronting gate confidence 0.05 to avoid false negatives, an embedding-based tool retrieval that narrows 13 tool candidates, and a Pydantic AI agent using native tool calling (Qwen 20B performed best) to cut latency and errors; benchmarks show ~97% tool selection accuracy, 0% false negatives in gate tests, and practical limits around history size and multi-tool retrieval.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:46Initial architecture
- 7:03Rebuilding with Pydantic AI
- 10:35Model selection
- 13:22Version two problems
- 16:55Gate component
- 23:17Tool retrieval layer
- 26:10Agentic loop
- 28:18Cost and demo
- 32:40Q&A
related talks
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
52mCut the Hype and Build Your Own AI Agents
41mGoing beyond copilot with AI Agents
25mBuilding Your own AI Assistant with JavaScript, LLMs, and Rag
36mProduction-Ready AI Agents: From LLMs to Small Language Models
36mBuilding Agentic AI Assistants: Lessons from productionising agents in Preclinical R&D
46mFullstack AI: The Building Blocks for Production Ready AI-Powered Software
38mBuilding your own Multiskill AI Assistant with DeepPavlov