Beyond Static Benchmarks: Chaos Based Agent...
summary
Claims an open source AgentCert (ACE) framework that runs Litmus Chaos-driven fault injections, collects telemetry with LangFuse, and certifies agents by repeating experiments (dozens of runs) against real apps like Sock Shop. Demo evaluates an SRE AIOps agent: fault detection around 90% but mitigation rate 2%, exposing a big gap between detection and automated recovery. The repo templates experiments, produces PDF certificates, and shows why static benchmarks miss brittleness that causes ~40% of projects to fail.
AI-written summary · about
chapters (10)
- 0:00Agent evaluation frameworks
- 1:24Levels of autonomy
- 7:26Enterprise challenges
- 12:01Trust in autonomous systems
- 14:25Rigorous evaluation frameworks
- 18:23Chaos engineering for resilience
- 20:09ACE compliance engine architecture
- 26:50ACE framework demo
- 34:43Agent certification reports
- 38:16Q&A
related talks
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
27mFrom Chaos to Confidence: How SREs Can Leverage 50 (and Counting) Failure...
23mThe Art & Science of Benchmarking Agents
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications
46mAI Testing Beyond the Basics: Ensuring Truthful and Reliable Agents
31mObservability in the Matrix: Choosing the Right AI Agent To Escape the Chaos
53mRapid Threat-to-Test for Agents
36mTalking Chaos: An AI Co-Pilot for Resilience With LitmusChaos