Evidence-Based QA: Proving Quality with Agentic Test Runs and OpenSourcing How We Do It
summary
Evidence-cli creates a sealed, versioned .evidence pack per run that contains L0 (run, test file, result) and optional L1 captures (per-step screenshots, network, console and failure records) and hashes the test definition so you can answer “is this the same test” mechanically. Commands are finalize, merge and validate returns 0/1/2 for CI, and the validator never reads test contents. Kane/ KCLI emits rich captures into the same open source schema repository so anyone can verify offline. agent recorded as author
AI-written summary · about
chapters (60)
- 0:00Start
- 0:08Session Opens - Evidence-Based QA
- 0:40"Passed" Is Enough Until You're in a Regulated Industry
- 1:10Which Test, Against Which Build, and How Do You Prove It?
- 1:42Speaker Intro - Anubhav Singhmaar, Product Manager, TestMu AI
- 2:14The Thing Every Team Produces Weekly and Throws in the Bin
- 2:46The Plan: Four Numbered Problems, Then the Fix
- 3:16A Quick Recap of What Kane CLI Is
- 3:46Kane Matters Here for One Unexpected Reason
- 4:20Being Fair to It - This Is Genuinely Good Software
- 4:50The Narrower Question: What Can It Not Tell You, Even in Principle?
- 5:21Problem One - The Report Is Stuck Inside One Tool
- 5:51A 2 AM Failure: Screenshot, Network, Element - All of It Existed
- 6:21Reduced to "Failed" Plus a Stack Trace, If You're Lucky
- 6:53It Cannot Notice That Someone Rewrote the Test in Build 14
- 7:23Problem Two - Everyone Builds Their Own Format
- 7:53The Real Cost Is Arithmetic: N Producers Times M Consumers
- 8:24An Industry of Smart People Competing on Reporting Instead of Bugs
- 8:54The Lock-In Nobody Sold You, Accumulated One Integration at a Time
- 9:24Problem Three - The Trust Model Broke
- 9:54"How Do You Know This Test Is Right?" Used to Have an Answer
- 10:27A Model Writes It, an Agent Runs It, an Agent Repairs It Overnight
- 10:57"Is This the Same Test That Passed Last Week?" Becomes Unanswerable
- 11:27Flakiness Detection and Trend Charts Silently Comparing Ghosts
- 11:59Problem Four - Someone Walks In and Asks You to Prove It
- 12:29Three Questions You Can Handle, Two That Silence the Room
- 13:00"Show Me Evidence for Last Quarter" - CI Logs Expired
- 13:31The Data Existed - Nobody Budgeted to Keep It
- 14:01Where All Four Problems Land: A Green Check Is a Claim
- 14:31Finance Has Ledgers, Aviation Has Flight Recorders, Security Has Audit Trails
- 15:01Introducing .evidence - One Sealed Archive Per Run
- 15:32L0: The Run File, the Test File, the Result File
- 16:04L1: Logs, Screenshots, Coverage and Failure Records Per Step
- 16:34A Reader You Write Today Still Works Against Future Profiles
- 17:04Closing Problems One and Two: The Pack Carries It Per Step, Integrate Once
- 17:35Closing Problems Three and Four: Hashed Definition, Sealed and Offline-Verifiable
- 18:05Evidence CLI and Its Three Commands
- 18:36validate and merge - Conformance Check With a Real Exit Code, Combining Sharded Runs
- 19:07What the Tool Deliberately Does Not Do
- 19:37It References Your Test File and Hashes It - It Never Reads It
- 20:09A Trust Artifact Cannot Be Owned by the Tool Being Judged
- 20:41Two Details That Save a Support Ticket: Producer Name and Step Ordinals
- 21:11One Shape for Browsers, Agents, API Checks and In-House Runners
- 21:41The Small Detail That Says Most: An Agent Recorded as the Author
- 22:12Agents Filing Issues Against the Format, With Their Own Templates
- 22:43Evidence CLI Is Open Source - and What's Actually in the Repo
- 23:14The JSON Schemas Are the Authority, Not the Docs
- 23:45Where Kane Sits: Consuming the Published Package, No Internal Fork
- 24:17What Kane Adds - The Captured Layer
- 24:5090% of "Why Did This Fail" Answered at a Glance
- 25:21Console Logs, Root Cause, Trajectory and Environment Fingerprint
- 25:52A Hand-Built Pack vs a Kane Pack: Same Seal, Different Depth
- 26:23The UI Renderer and Its Four Tabs
- 26:54Report, Test Instances, Issues and Requirement Coverage
- 27:26Knocking Off All Four Problems One by One
- 28:27A Green Check Mark Is a Claim; .evidence Is the Proof
- 28:58Q&A: Making Agent Run Results Trustworthy to an Outsider
- 29:29If Someone Edited the Test Afterwards, the Hash Stops Matching
- 30:00Q&A: What Counts as Evidence in an Agentic Test Run?
- 30:31The Minimum Three Things vs Everything in the Layer Above
related talks
47mEvolution of Quality Engineering in Financial Services
46mQA to QE: Scaling Quality with Ownership, Code, and Culture
46mAI Testing Beyond the Basics: Ensuring Truthful and Reliable Agents
37mAI agents testing How to evaluate the unpredictable
43mCan you Test it
23mBeyond Tests What to Verify in AI Generated Code
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
32mAQA: You Are Soaking In It