Anthropic Added Plugin Evals to Claude Code With Six Grader Types and a CI Gate That Measures the Delta a Plugin Actually Adds
MarkTechPost·high signal
Claude Code v2.1.269 ships an eval harness for plugins and skills with four free graders (regex, tool_used, tool_order, file_exists) and two paid ones (llm, baseline) that require judge model calls. It runs as a CI gate via --threshold, --max-cost-usd and --trust-plugin, and without --trust-plugin an untrusted checkout exits 1 when no terminal is attached. The design decision worth copying is that the headline metric is the delta between runs with and without the plugin loaded, not the absolute score, which is the only number that shows the plugin did anything.