Author metadata affects Large Language Model scores in scientific peer review

Abstract

Large language models (LLMs) are increasingly considered for scientific evaluation workflows, including editorial triage, screening, and ranking. This raises a fairness and reliability question: are LLM-based evaluators counterfactually invariant to extrinsic author-context metadata when manuscript content is held fixed? We conduct a controlled counterfactual audit of 80 recent English-language arXiv manuscripts from computer science, mathematics, physics, and quantitative biology. For each manuscript, we append a corresponding-author profile in a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2\times 2\times 2$$\end{document} factorial design varying name-coded identity, institution prestige tier, and bibliometric profile, alongside a no-metadata BLIND baseline. Six heterogeneous LLM evaluators assign ratings on 14 review-style criteria and three summary outcomes: content quality, perceived prestige/impact, and acceptance recommendation. Estimates are computed at the manuscript level after averaging repeated calls within model and pooling evaluator models with equal weight; uncertainty is assessed with bootstrap confidence intervals, sign-flip tests, false-discovery-rate control, and model heterogeneity analyses. Evaluator outputs are not invariant to author metadata. Bibliometric and institution-tier cues systematically increase perceived prestige/impact and acceptance-related judgments, while effects on content-quality judgments and name-coded effects are smaller. Ranking analyses show that metadata perturbations can displace manuscripts and alter top-K shortlist membership. These results identify a concrete reliability risk for LLM-assisted scientific evaluation pipelines.