Cybersecurity LLM benchmarks swing 80+ points on scoring setup

Researchers auditing eight cybersecurity benchmarks found that a single measurement decision, not model capability, can move a score by more than 80 percentage points. The reliability audit of eight cybersecurity benchmarks treated each benchmark as a configurable measurement setup instead of a fixed dataset. It tested that setup across 10 proprietary, open-weight, and security-specialized models.

That matters for anyone picking a model off a leaderboard. The paper documents 15 systematic failure modes tied to these setup choices, and two task pairs that are semantically similar rank the same models in different orders purely because their evaluation conventions don't match.

The team then built a harness that standardizes those choices while keeping task semantics intact. Under it, nine of the 10 models moved at least three ranks on at least one benchmark. A model that topped the original leaderboard can fall out of the top tier once the scoring setup changes, with nothing about the model itself different.

The paper doesn't say which models moved the most or which of the 15 failure modes caused the most damage. It's not a guide to which vendor benefits from loose scoring. What it establishes is that a leaderboard number carries a hidden configuration nobody wrote down.

Anyone using these benchmarks to justify a security-model purchase or an internal rollout is trusting a number without knowing what setup produced it.