The findings suggest that modern Large Language Models can provide useful and consistent support for scientific peer review, however remaining differences between AI-generated and human-generated evaluations indicate that current systems should be viewed as complementary tools that assist human reviewers rather than replacements for expert human judgment.
It is demonstrated that aggregate quality scores alone can overestimate review quality and argued for multi-dimensional evaluation of AI-generated peer reviews.
Alexander M. Fichtl, Lukas Ellinger, Josefin Kelber et al.· 1 citation
Reviewerly's retrieval-centered infrastructure for auditing peer review at scale is presented and three deployed systems are described, including three deployed systems that analyzes hybrid human--AI authorship by disentangling the origin of ideas from the origin of text in peer reviews, enabling measurement of how hum...
Negar Arabzadeh, Sajad Ebrahimi, Alireza Daghighfarsoodeh et al.· Annual International ACM SIG...· 1 citation
Large language models are increasingly used for research quality evaluation, with prior work exploring their scoring accuracy and the plausibility of review rationales exploring their scoring accuracy and the plausibility of review rationales.
AI cannot replicate the nuanced judgment of human peer reviewers at a high-impact sports medicine journal and has potential utility as an exploratory first-pass screening tool warranting further validation in larger cohorts.
Romir Patel, Christopher L. Shultz, Mathieu Ollivier et al.· American Journal of Sports M...· 0 citations
The results raise concerns that LLM-assisted evaluation may under-select proposals that human reviewers identify as highly novel, potentially reflecting the statistical logic of next-token prediction trained on past scientific outputs.
Background: Large language models (LLMs) are increasingly used to automate code review, but the reasoning behind their decisions remains hard to understand. Developers struggle to assess the validity of LLM-generated reviews, making it difficult to gauge how much trust to place in them. The role of Explainable AI (XAI)...
Zhenhan Gao, Marvin Muñoz Barón, Umm E. Habiba et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
