The Judge Has Biases

Position bias — in pairwise comparisons, judges systematically prefer the first (or last) response presented. Same outputs, swapped order, different winner — the verdict reflects position, not quality.

Verbosity bias — longer, better-formatted answers score higher even when the extra length adds nothing; a padded non-answer beats a terse correct one.

Self-preference — a judge favors outputs resembling its own style: GPT-family judges prefer GPT-family outputs; the grader has a taste, and the taste is itself.

None of these are subtle. All three are documented, reproducible, and big enough to flip A/B verdicts.