'When the Judge Changes, So Does the Measurement' Shows LLM-as-Judge Upgrades Aren't Interchangeable

arXiv·high signal

This paper treats evaluator-replacement as a measurement-validity problem: an LLM-as-judge score can move even when candidate responses are held fixed, simply because you swapped the judge. Across four judgment datasets it compares two real upgrade paths — scaling Qwen3 dense judges from 1.7B to 32B, and moving across MiniMax M2 to M2.7 APIs — and finds judge upgrades are not interchangeable (only Qwen3 1.7B-to-4B gave a robust adjacent gain). A caution for any team that silently upgrades its eval judge and assumes scores stay comparable.