Harnesses in AI: A Deep Dive
summary
Demonstrates that harness engineering not prompt tuning makes GPT-3.5 Turbo reliably log in and upvote on Hacker News. Builds a Playwright browser agent, adds guardrails (max iterations, context compression), a verify step that inspects tool-call history to detect lies, and a login handler that injects credentials programmatically. Caps attempts to three, uses deterministic checks, and ends with a reproducible successful upvote after the harness fixes failures.
AI-written summary · about
chapters (13)
- 0:00Introduction to Tejas Kumar and AI Harnesses
- 1:45Why we use harnesses: Reliability and control
- 3:00Defining an agent harness from first principles
- 4:32Key components of an agent harness (Tooling, Context, Guardrails)
- 5:59Starting the demo: Building a browser agent
- 7:00Inspecting the initial agent loop
- 8:12The problem: Agent failure and hallucination
- 10:20Adding guardrails and context management
- 11:54Refactoring into a formal harness
- 13:02Implementing a verify step to catch lies
- 15:36Implementing a login handler for programmatic access
- 17:42Final demonstration: Successful autonomous upvoting
- 18:34Summary and the future of dynamic harnesses
related talks
20mHow Will We Prompt AGI? A History of Harness Hacks
32mWhat if the harness mattered more than the model?
21mConvention Over Hallucination: Harness Engineering for AI-Powered...
36mHow NOT to Train Your Hack Bot: Dos and Don'ts of Building Offensive GPTs
31mEngineering the harness
25mAttack, Defense & Danger in the Age of AI
01h 16mAutomation of the Automation using ChatGPT
37mGenerative AI: Beyond (Just) Hype