PREreview of "(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection"

Abstract

This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/23335998. Thank you for this paper. You build CRBugBench from human-verified bugs mined from repository history and traced back to the pull request that introduced them, 1,000 tasks from 98 open-source GitHub repositories plus 800 tasks from two proprietary industry repositories (Sections 3.2-3.5). The agent reviews the change in a sandbox without being told a bug exists, and three LLM judges match its self-ranked findings to the known bug (Section 3.3). The best configuration, claude-opus-5 with copilot-cli, catches 71.9% of bugs in its top-5 findings, up from 47.7% for opus-4.6, and performance drops by up to 25.5% on the proprietary repositories (Section 4.3). I liked that ground truth comes from bugs humans confirmed and fixed, and that you report the combined result: 50% for humans alone, 71.9% for the best agent, 82.1% together. I read it as a practitioner who runs coding agents every day. Four comments. 1. On reading versus running - In Section 4.5 you show that reading and searching take 75% of turns, while running code takes about 8%. claude-opus-5 executes code in 9.7% of turns and gpt-5.6-sol in 6.9%. You also describe the overtrust cases: in the Selenium task all models endorsed the buggy condition because the in-code comment and the PR description asserted the intended behavior, and in the Elasticsearch task opus-5 cited the defective line as its standard of correctness. In my practice, a check the agent can run is the only signal other than "looks done", and the agent sounds equally sure when it is right and when it is wrong. Could you split recall@5 by whether the trajectory contained at least one Execute action, per model? And were findings that say the code was run more often correct? 2. On how the second agent is briefed - in the multi-agent setups you test (Section 4.1), the second agent is asked to improve the first pass (self-refine) or to consolidate three or five candidate reviews (candidate-ensemble). In my practice the second, fresh agent gets a different task: disprove the first one's work and treat it as wrong until shown otherwise. Independent agents rarely make the same mistake, so where they disagree is usually the weak spot. A round of checking lowers the risk but doesn't remove it, which fits your Section 5.1 conclusion that an accountable human keeps the final decision. Given that multi-agent setups add +2.8% on average at 3.68x cost (Section 4.6, Table 3), did you try a skeptic brief, where the second agent must refute each finding of the first, or must find a bug the first review missed? And in the mixed ensemble, is disagreement between candidates on a file a useful signal of where the bug is? 3. On where the cost goes - Section 4.3 reports $2.97 per task for claude-opus-5, and Table 1 the tokens. In my logs of 722 coding-agent sessions, more than 85% of modelled spend was context work, meaning cache reads plus cache writes, and sessions longer than 200 calls, 8% of sessions, gave more than 90% of the money (DOI 10.5281/zenodo.22759216). These are sessions I ran myself, with cost modelled from list prices, not invoices. So the half-life gap in Section 4.5 (turn 17 for gpt-5.6-sol, turn 35 for claude-opus-5) matters for cost. In my practice heavy reading goes to sub-agents, so the main conversation stays clean. Could Table 1 or an appendix show the cache share of cost per task, and cost per task against number of turns? For the claude-code runs in Section 4.6, does the cost increase come from the sub-agents' own contexts or from the main loop? 4. On data - You say the replication package can't be made public, and that an open-source version of the benchmark is on GitHub (malusamayo/CRBugBench). For that open-source part, could the release include the review.md files per model, the per-finding judge votes, and the tool-call traces? Then readers could check the questions above without rerunning ten models. For comparison, the run data of my 36 coding-agent runs is public on Hugging Face (DOI 10.57967/hf/10366) and on OSF (DOI 10.17605/OSF.IO/5QTWY), so anyone can recompute it. These are open data records, not peer-reviewed publications. Thank you for the careful write-up. Competing interests The text cites the author's own technical report (DOI 10.5281/zenodo.22759216) and open dataset (DOI 10.57967/hf/10366, DOI 10.17605/OSF.IO/5QTWY); no connection to the article's authors. Use of Artificial Intelligence (AI) The author declares that they used generative AI to come up with new ideas for their review.