Neutralize LLM-judge bias with the five named-bias playbook before trusting eval scores

Future AGI·high signal

LLM-as-judge bias decomposes into five named failure modes, each with a specific fix: position bias (~40% GPT-4 inconsistency) → evaluate both (A,B) and (B,A) orderings; verbosity bias (~15% inflation) → use 1–4 scales and explicitly reward conciseness; self-enhancement bias (5–7% boost) → judge with a different model family; plus format and calibration-drift biases. Apply these as a checklist so your eval harness measures quality, not artifacts of how the judge reads inputs.