Skip to content

Detecting Self-Preference in a Single LLM Judge

Self-preference in an LLM judge shows up only when you split verdicts by the generator's model family; a pooled agreement score hides it.

Measure your judge's verdicts separately for candidates its own family produced and candidates other families produced. Self-preference is a conditional error: it lives in one of those strata and is absent from the other, which is why pooling them hides it. Stratify first, then compare.

When this is worth measuring

The check earns its cost under narrow conditions, and skipping it is often correct.

  • The eval has no checkable ground truth. Where tests pass or an answer matches exactly, execution settles the question and the judge is not the instrument.
  • The judge and at least one generator share a model family. Otherwise there is no self stratum to compare against.
  • You have enough items in each stratum to compare them. See the variance floor below.
  • Verdicts gate something. A judge that only ranks experiments for a human to read tolerates a bias that a merge gate does not.

How to detect it

Hold the candidates fixed and vary only the judge. Score the same set twice, once with a judge from the generator's family and once with a judge from a different family, then compare pass rates within each stratum. A judge that scores same-family output higher than the cross-family judge does, while the cross-family judge agrees with humans on both strata, has self-preference. One practitioner report describes that shape: after swapping the generator to a different model, "the judge got noticeably stricter. It caught issues in the other model's queries that it had been letting through" (Towards Data Science, 2026).

Permute output order under the same design. Position and verbosity bias were named alongside self-enhancement bias in the original study of LLM judges, which also found strong judges reach over 80% agreement with human preferences (Zheng et al., 2023, arXiv:2306.05685v4). A flip rate under permutation separates position bias from the family effect.

Why it works

Two published mechanisms both predict a conditional error, and neither predicts a uniform one. Under the recognition account, judges identify their own generations and favor what they recognize: GPT-4 is 73.5% accurate at distinguishing itself from two other LLMs and humans out of the box, and fine-tuning GPT-3.5 to over 90% self-recognition raised the example-level correlation between recognition and preference from 0.41 to 0.74 on XSUM (Panickssery et al., 2024, arXiv:2404.13076v1). Under the familiarity account, the judge rewards fluency rather than authorship: across eight LLMs, GPT-4 scored highest on a self-preference metric at 0.520, yet the same analysis found that "across all models, except for dolly-v2-12b and stablelm-tuned-alpha-7b, no significant difference was observed between the judgment rates for their own texts and those generated by other models", alongside judges rating low-perplexity text above what human evaluators gave it (Wataoka et al., 2025, arXiv:2410.21819v2). The accounts disagree about the cause and therefore about the fix, which is why the ablation comes before the correction.

What the corrections actually buy

Every standard correction is partial. The measured limits matter more than the recommendation.

Correction Measured limit
Swap to a cross-family judge Judge accuracy tracks the judge's own task accuracy at Pearson r of 0.795, 0.708, and 0.899 on MATH500, MMLU, and MBPP+, so a weaker neutral judge can cost more accuracy than the bias it removes (Chen et al., 2025, arXiv:2504.03846v3)
Add a panel of diverse judges A panel of smaller disjoint families runs "over seven times less expensive" than one large judge (Verga et al., 2024, arXiv:2404.18796v2), but nine judges from seven families supply about 2.18 independent votes, and the best single judge matched or beat the panel in every condition tested (Kohli, 2026, arXiv:2605.29800v1)
Blind the candidate's provenance Light perturbation such as synonym replacement reduces self-preference, but under fuller neutralization of stylistic differences "self-preference recovers" (Mahbub and Feng, 2025, arXiv:2512.05379v1)
Calibrate against a human golden set Catches aggregate judge error, not this one. It is the practice described in meta-evaluating the judge, and it averages the two strata

When this backfires

  • The task has ground truth. On verifiable benchmarks a capable judge's self-preference is mostly justified: "Qwen-2.5-70B and Llama-3-70B achieve LSPR of 96.57% and 95.16% on MATH500, respectively", where LSPR is the share of a judge's self-preferences that were objectively correct (Chen et al., 2025, arXiv:2504.03846v3). Correcting there removes signal.
  • The eval set is small, or the judge sits outside the window where a permutation diagnostic resolves anything. "Position bias turns out to be statistically detectable only within a roughly 60–95% base-accuracy Goldilocks zone", and above that band "the absence of bias signal in those cells should be read as 'not measurable on this instrument,' not 'model is unbiased'" (Tamba, arXiv:2607.20864v1). A clean result there is not a pass.
  • You correct by scaling the panel. Frontier judges err on the same items, at a mean pairwise phi correlation of 0.391, and better aggregation closes at most 11% of the resulting accuracy gap even with gold labels (Kohli, 2026, arXiv:2605.29800v1).
  • The judge is the strongest model available. Harmful self-preference concentrates where that judge errs: Qwen2.5-72B preferred its own wrong answer 86% of the time on MATH500 and 73% on MMLU (Chen et al., 2025, arXiv:2504.03846v3). Downgrading the judge to escape that trades a narrow failure for a broad one.

Key Takeaways

  • Stratify verdicts by the generator's model family before reading any agreement number; one pooled figure hides a conditional error by construction.
  • The detection is a judge swap with the candidates held fixed. A rubric rewrite measures nothing.
  • No published correction removes the bias. Pick the one whose measured limit you can live with, and record which one you accepted.
  • On tasks with checkable ground truth, take the judge out of the loop rather than correcting it.
Emerging — single recent source Last reviewed: 2026-08-21