Does positional bias in LLM-as-judge evaluation, and its calibration fix, hold for open-weight judges?

Abstract

A replication of Wang et al. (2024), "Large Language Models are not Fair Evaluators", under judge substitution. Wang, Peiyi; Li, Lei; Chen, Liang; Cai, Zefan; Zhu, Dawei; Lin, Binghuai; Cao, Yunbo; Kong, Lingpeng; Liu, Qi; Liu, Tianyu; Sui, Zhifang. "Large Language Models are not Fair Evaluators." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440-9450, 2024. DOI: 10.18653/v1/2024.acl-long.511. Original code and data: https://github.com/i-Eval/FairEval The original study used two closed OpenAI endpoints as judges. It reports: "We use the OpenAI API to conduct our experiments ('gpt-3.5-turbo' for ChatGPT, and 'gpt-4' for GPT-4)." The snapshot pinned in the original repository, gpt-3.5-turbo-0301, has been retired. The original numbers therefore cannot be reproduced by anyone, including the original authors. This study repeats the published protocol while varying one technical detail deliberately - the judge model - and asks: Do the positional-bias finding and the MEC/BPC calibration recovery reported by Wang et al. (2024) hold when the judge is a 2026 open-weight instruction-tuned model instead of a 2023 closed OpenAI endpoint? This is a protocol-generalization replication under judge substitution. It is not a claim to have reproduced the original's absolute numbers, and it is not framed as one. Claims under test, quoted from the published paper: C-1 Positional bias exists and is large. Original Table 2: "GPT-4 Vicuna-13B v.s. ChatGPT 51.3% 23.8% 37 / 80 (46.3%)"; "GPT-4 Vicuna-13B v.s. Alpaca-13B 92.5% 92.5% 4 / 80 (5.0%)"; "ChatGPT Vicuna-13B v.s. ChatGPT 2.5% 82.5% 66 / 80 (82.5%)"; "ChatGPT Vicuna-13B v.s. Alpaca-13B 37.5% 90.0% 42 / 80 (52.5%)". Conflict rate is defined there as "the proportion of conflicting results given by the same evaluator when simply changing the position of two models." C-2 Bias direction is evaluator-specific: "on the same evaluation template T in Table 1, GPT-4 tends to favor the response in the first position, while ChatGPT shows a preference for the response in the second position." C-3 Conflict rate depends on the quality gap: "The conflict rate is negatively correlated with the score gap between the two responses. When swapping the order of two responses, the smaller the score gap between them, the more likely GPT-4 is to produce conflicting results." C-4 The calibration recovers alignment. Original Table 5, ChatGPT as evaluator: "SCORING VANILLA 44.4% 0.06 82.5% | SCORING MEC 53.2% 0.24 35.0% | SCORING MEC + BPC 58.8% 0.31 N/A | COMPARING VANILLA 50.2% 0.18 50.0% | COMPARING MEC 54.8% 0.27 42.5% | COMPARING MEC + BPC 60.0% 0.35 N/A" (accuracy, kappa, conflict rate). C-5 The human ceiling, used as a yardstick and not itself under test: "the average accuracy and the kappa correlation coefficient of human annotations are 71.7% and 0.54, respectively".