Herding LLMs: Predictable Outcomes from Unpredictable Agents
summary
Claims that event-sourced, durable workflows plus domain engineering and code-level guardrails turn chaotic LLM agents into reliable services, demonstrated by AKA runtime running a multi-node agent workflow that fails over mid-flow and resumes on another node successfully. Patterns include durable workflows, schema-validated inputs/outputs, circuit breakers and human escalation, plus adaptive policies that auto-switch models (moving fixes from days to minutes). Concrete pains: a two-week time-based bug at a client, token-loop failures, and observable signals for automated EvalOps.
AI-written summary · about
chapters (19)
- 0:00Chatbots to autonomous agents
- 3:04Stochastic vs non-determinism
- 6:37Language trap in agent communication
- 9:28Event-driven systems foundation
- 12:50Domain engineering for AI
- 15:49Three pillars of LLM execution
- 20:05Mining agent signals for data
- 22:11Adaptive policies as guards
- 27:16Durable agent workflows in practice
- 30:28Demo: workflow resilience
- 36:07Guardrails and schema enforcement
- 38:47Demo: guardrails blocking unsafe requests
- 42:13Dynamic routing and human-in-the-loop
- 45:01Multi-agent debate and collaboration
- 46:24Compositional reasoning for tasks
- 48:10Handling exceptions safely
- 50:40Tracking agent decisions
- 52:27Beyond evalops to self-improving
- 55:14Summary and architecture
related talks
20mBuilding Deterministic Infrastructure for Non Deterministic AI Agents
25mFrom Chaos to Control: Building Enterprise-Grade LLM Agents That Actually Work
01h 03mAgents Everywhere
32mEvent-Driven & Multi-Agent Architectures for Gen AI Workflows
50mHow to Test LLM Agents
01h 00mFrom LLM Theory to Practical Agentic Implementations
26mContext Engineering: Practical Techniques for Improving Agent Quality Today
26mHow Bad Actors AbUse LLMs The Mechanisms That Enable Misuse and What We Can Do About