Large language models and human judgement in scientific peer review: evidence from over 1000 published economics papers

Abstract

The peer‐review process in economics faces two well‐documented challenges: a shortage of willing referees, and prolonged turnaround times. We examine whether large language models (LLMs) can help to address these inefficiencies without introducing systematic bias, using two pre‐registered experiments across four LLMs, and over 29,000 evaluations of 1220 anonymized papers. LLMs reliably differentiate research quality by journal rank, and ratings are positively correlated with realized citation impact. However, model performance varies substantially: GPT and Gemma show the strongest evaluative gradients, while LLaMA assigns the maximum score to 81% of papers, rendering it uninformative for quality discrimination. GPT excels at detecting AI‐generated content; Claude‐generated papers receive the highest quality ratings and are the hardest to detect. Varying author identity in the second experiment reveals that three of the four LLMs assign significantly higher ratings to papers attributed to prominent economists and elite Western institutions, and lower ratings to papers with non‐Western affiliations, mirroring the prestige gradients documented in the human peer‐review literature. These biases persist when paper quality is held constant, highlighting the necessity for author‐blind inputs for any editorial deployment of LLM‐assisted screening.