Calibrate LLM judges to Cohen's kappa ≥0.6 and test for five named bias types before trusting them

Digital Applied·high signal

Before gating on a model-based grader, calibrate it against 100+ human-labeled examples to a Cohen's kappa ≥0.6, use binary pass/fail verdicts instead of Likert scales, and explicitly test for position bias (~70% first-response favoritism), verbosity bias (>90% preference for longer answers), self-preference (+10–25% same-family inflation), format bias, and calibration drift. Recalibrate monthly and keep judge cost under 10–15% of production LLM spend. Most teams skip calibration entirely and ship judges that quietly reward the wrong thing.