Typos Make LLM Judges See Bias That Isn't There, Flipping Neutral to Biased up to 120x More Often Than the Reverse
arXiv·low signal
arXiv 2609.11067 applied five realistic noise types at several intensities to 3,822 stereotype-related responses and compared four LLM judges' bias ratings before and after. The noise pushed neutral judgments to 'biased' up to 120x more often than it pushed biased judgments to neutral, and the effect was purest at mild, realistic noise levels. Bias audits run on user-generated text with an LLM judge will systematically overestimate bias.