AgentAudit attaches to a running agent and scores its trace on ten dimensions, exposing 95.1 vs 22.6 trust spreads at similar task completion

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.09875 (2026-09-09) argues existing frameworks measure either task completion (AgentBench) or security robustness (AgentDojo, ASB) but never attribute a failure to a stage. AgentAudit reads only the recorded execution trace — imposing no constraint on the agent's implementation — and scores instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security and execution integrity, then classifies behavior and attributes the failure to a specific stage. Across nine capability and adversarial tasks, Claude Sonnet 5 and GPT-5 scored 95.1 and 80.6 Composite Trust while Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash scored 57.6, 45.7 and 22.6. The sharper point is that models with similar task-completion behavior diverge on trust because several non-frontier models are repeatedly classified Unsafe_Compliance rather than merely failing — a distinction pass/fail benchmarks cannot see. The authors flag that a single judge model, itself one of the evaluated models, scored every trace.