PREreview of "An Empirical Study of Harness Design for Coding Agents"

Abstract

This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/22879230. The authors take a coding harness apart and vary planning, the action space and context management over four models, two benchmarks and 176 matched settings. They report that context management is worth most when the window is tight (it prevents overflow terminations), that elision staged before summarization (T4) is the cheapest strategy at similar accuracy, and that planning is an accuracy scaffold for the weakest model but a cost saver for stronger ones. I read it as a practitioner who runs coding agents every day, not as a benchmark author. In our company the software development is done by AI agents and I measure what they bill me, so maybe my practical side is useful here. My comments are about money and verification. 1. The cost column is the most useful thing here, and I would like it split. Table 3 and Table 4 give mean cost per task, Table 5 the change in input tokens per call. In my measurement of 722 agent sessions (150,902 model calls, 34.6 billion tokens) more than 85% of the modeled spend was work with context - cache reads plus cache writes - not generation (DOI 10.5281/zenodo.22759216, technical report, not peer-reviewed). On a separate 168-session run, 94.4% of paid tokens were re-reads of context already sent (DOI 10.5281/zenodo.22712985), a share of tokens and not of the bill. Rewriting the prefix is charged too, so a policy that looks cheap in M3 calls (Figure 5(b)) can still push money into re-sent context. Logging cached input, fresh input and output apart would show whether T4 wins by calling the summarizer less or by carrying a shorter prefix each turn. 2. Your managed-T0 gap matches something I got from another direction. In 3.2 it falls from 35.7 to 2.7 percentage points on SWE-Bench between 32k and 128k, while T4 keeps the lowest mean cost at every budget. I ran an experiment on how often to clear an agent's context between tasks - 36 runs, six session-length policies, six repeats each, 4,086 tests. The curve was U-shaped, the minimum sat at three tasks per session, three to six were a plateau, and clearing after every task came out about a third more expensive (+33.5%, p = 0.002, 0.011 after Holm correction; DOI 10.5281/zenodo.22759217). Two caveats on my number - the comparison point was picked after the runs, and in the clear-every-task arm the agent wrote more tests, so part of the gap may be extra work. 3. About planning cutting post-edit verification - I would be careful with that outside the benchmark. Section 4 reports that planning drops the median SWE-Bench trajectory from 108 to 74 turns for Nemotron-3 550B and from 68 to 53 for Mistral-Medium-3.5-128B, with most of the cut in the Verify phase. On SWE-Bench a hidden test suite grades the run afterwards, so the verification the agent skips is done for it. In my work nobody grades the run, and my rule is that every task needs a check the agent can run itself, because otherwise "looks done" is the only signal it has. The model sounds the same when it is right and when it is wrong (a made-up fact, a missed case and a "fixed" bug arrive in one confident tone). So I read that as stopping behaviour under an external grader, and Table 6 counts runs that stopped without an edit, not runs that stopped without verifying an edit they did make. 4. The planning ablation changes the prompt as well as the component. Section 2.1 with Figures 15, 16 and 17 shows planning arrives as a system block, a first-turn reminder, and the plan re-inserted every later turn, so T4 w/o plan drops an instruction block together with the plan state. I keep agent instruction files short in my own work (around 200 lines, and the test for each line is whether the agent starts making mistakes if I delete it) because long files get followed worse. A length-matched control, with the planning block swapped for inert text of the same size, would show how much of the +293.2% turns for Nemotron-3 30B in Table 5 is the plan and how much is more instruction. Competing interests Yes: the text cites the author's own technical reports (DOI 10.5281/zenodo.22759216, 10.5281/zenodo.22759217, 10.5281/zenodo.22712985); no connection to the article's authors. Use of Artificial Intelligence (AI) The author declares that they did not use generative AI to come up with new ideas for their review.