Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review
TL;DR
Results show that model rankings depend on the evaluation protocol: claude‐haiku‐4.5 achieves the highest adjusted overall score in the human evaluation, whereas gpt‐5‐mini achieves the highest aggregate scores in the automated LLM‐as‐judge evaluation and the lowest estimated operational cost.
Abstract
This study proposes an AI‐assisted peer‐review framework based on retrieval‐augmented generation (RAG), advanced prompt engineering and multi‐agent large language model (LLM) orchestration to support human reviewers and editorial decision‐making through structured, evidence‐aware manuscript assessment. Manuscripts are represented as structured analytical objects connected to vectorized corpora of Q1/Q2 journal literature and reviewer guidelines. The framework employs five specialized reviewer agents: structural, methodological, incremental novelty, disruptive novelty and claim verification reviewers, using role‐conditioned and retrieval‐grounded prompts to produce schema‐constrained analytical outputs. These outputs are synthesized through a meta‐review before being delivered to a human reviewer. The system is evaluated on 100 manuscripts. AI‐assisted reviews are generated with gpt‐5‐mini , claude‐haiku‐4.5 and gemini‐2.5‐flash , while review quality is assessed through blinded human evaluation and an automated LLM‐as‐judge procedure implemented with grok‐4.3 . Results show that model rankings depend on the evaluation protocol: claude‐haiku‐4.5 achieves the highest adjusted overall score in the human evaluation, whereas gpt‐5‐mini achieves the highest aggregate scores in the automated LLM‐as‐judge evaluation and the lowest estimated operational cost. Retrieval‐grounded prompts produced small but consistent gains across the six evaluated quality dimensions, including a 0.10 increase in adjusted_overall_score , while the no‐RAG variant remained highly competitive.