The Production AI Playbook: Deploying Agents at Enterprise Scale
summary
A retail bank spends £85,000 PoC over six months failing to reach production until a team builds an evaluation dataset, tracing infrastructure and a measurement pipeline, selects the model picked in week seven, and launches; six weeks post‑launch observability traces reveal stale embeddings caused wrong answers after a policy update. The five pillars are evaluation, observability (trace every agent decision), data foundation, multi‑agent orchestration, and governance — a production incident playbook works and detects 47 PII breaches caught in testing.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:51Demo to production lessons
- 3:15Critical gaps in AI production
- 4:43Five pillars of AI playbook
- 5:03Pillar one: Evaluation metrics
- 9:17Three layers of AI evaluation
- 12:25Pillar two: Observability and tracing
- 15:15Pillar three: Data foundation
- 19:51Centralized tracking strategy
- 20:12Pillar four: Multi-agent orchestration
- 22:40Pillar five: Governance and risk
- 24:24Databricks Agent Bricks overview
- 24:45Retail banking chatbot case
- 31:02AI incident playbook
- 32:47Actionable steps
- 33:29Three real project lessons
- 36:19Resources and closing
related talks
08mProduction Evals For Agentic AI Systems
18mFrom POC to Profit: Deploying Customer-Facing AI Agents That Actually Work
44mPanel Discussion: The BFSI Playbook - Scaling AI with Trust and Quality
34mThe Hard Truth About AI Agents in Production
07h 37mAI Agents Conference 2026 - Day 3
19mTrusting Agents With Capital: The Gearbox Playbook
24mFrom First Experiments to AI Agents: Adoption Experience and Future Challenges
22mThe Agent Control Plane: Managing Thousands of AI Agents