The Agent-Eval Checklist: stop letting your evaluator leak answers and run untrusted code

AI Agent Insights·medium signal

A 2026 reliability study (4.49M tests across 6,259 production agents, 56.6% aggregate success) produced a hard checklist most teams violate: isolate the agent from the evaluator, never pass reference answers into the agent's context, never eval() untrusted model output, sanitize LLM-judge inputs against injection, and adversarially test the evaluator before trusting its scores. The skill: audit your eval harness against these five rules — a leaking or injectable evaluator silently inflates pass rates and hides real failures.