Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops
summary
Live workflow shows concrete changes: run evals against a golden dataset (65 quick cases, 650 full MPs) and measure accuracy rising from ~85% with a simple prompt to 92% with a tuned prompt and 96.7% after GEPA optimization. The demo uses Pydantic AI agents, Logfire gateway (Anthropic/OpenAI/GPT‑4.1), and Logfire managed variables to edit prompts, models and rollout percentages in a FastAPI server without redeploying; proposer agents iterate prompts, cost shows up (a few dollars during runs) and traces record each candidate.
AI-written summary · about
chapters (12)
- 0:00Introduction to Samuel Colvin and the Pydantic ecosystem
- 1:29Overview of GEPA for prompt optimization
- 3:02Introduction to Logfire managed variables
- 3:55Case study: Analyzing political dynasties using Wikipedia data
- 10:04Getting started: Setting up the environment and API keys
- 16:55Running the initial evaluation (evals) against a golden dataset
- 25:16Comparing different prompt performance
- 34:00Running the full GEPA optimization process
- 43:43Q&A: Handling prompt size and systemic errors
- 57:01Demonstrating managed variables in a FastAPI web server
- 1:11:06Discussing implicit user feedback collection
- 1:15:42Q&A: Real-world internal use cases and context engineering
related talks
01h 04mBuilding AI Applications the Pydantic Way
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
01h 26mShipping AI That Works: An Evaluation Framework for PMs
14mPrompt Engineering is Dead
33mAutomatic Agent Optimization and the Rise of Self-Improving Agents
01h 53mAgentic AI: Build a Multi-Agent Application with CrewAI
01h 12mChatGPT and OpenAI API solutions: 👑 My Favourite Implementations
52mCut the Hype and Build Your Own AI Agents