Skip to content

Merged, Not Measured: An Empirical Study of Performance Issues Fixed by Coding Agents

Sep 2026 · arXiv.org · Vol abs/2609.37985 · 0 citations
Computer Science

Abstract

Coding agents open pull requests (PRs) that claim to speed up software, but studies of human performance fixes say little about how maintainers respond to such a fix or whether its claim holds. From the 71,677 agent PRs of AIDev v4, a text filter and codebook coding by language models and by the authors select 1,262 performance issues fixed by six agents in 582 repositories. We code each issue and its tests and re-execute 23 rejected and 30 merged fixes. (1) 57% of closed fixes are merged, 61% of rejections give no stated reason, and only 6 of the 23 re-executed rejected claims held under our three-run pilot on mostly agent-built workloads. (2) Acceptance rises with the agent's track record in the repository (31-37% to 70%) and with the repository's pre-opening merge rate on its other agent PRs (33% to 84%). Merged fixes delete a larger share of the lines they change (0.26 versus 0.15), a difference that holds within agent and within repository, with no such difference detected in the coded content, description, tests or measurements. (3) Repeated computation and redundant data processing cause 44% of the issues, and 46% of fixes are architectural-level. (4) Agents change tests in 37% of fixes and 11% carry a performance test or benchmark; of the 30 merged fixes, 18 met our delivery criterion, 3 fell short of the claim, 9 showed no significant gain or regressed, and 14 change behavior on untested inputs. The outcome tracks the repository's history with the agent rather than the coded content of the fix, and a merge does not show that the fix delivers what it claims.

View source

Similar papers

Review Sep 2026

Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild

This paper studies 37,623 provenance-labeled pull requests from five commercial agents and combines the AIDev dataset with 58,792 cached GitHub API responses to measure security smells in added code, structural maintainability, post-merge churn, revert rates, and human review behavior.

Obada Kraishan · 0 citations
Preprint Oct 2026

Hidden in Plain Sight: an Empirical Study about The Instruction Stacks of Coding Agents

A longitudinal, cross-agent study of the instruction stacks captured in the first model requests of 1,886 releases of Claude Code, Codex, Gemini CLI, Qwen Code, and OpenCode, exposing an evolving, incompletely documented layer of agent design.

Felipe de Sant'Anna Paixão, Dhyego Tavares, Eduardo Almeida et al. · 0 citations
Preprint Oct 2026

Beyond Benchmarks: Can Small Open-Weight Models Get Real Coding Work Done?

Making an open-weight language model generate text is not the same as making it finish coding work. This retrospective empirical systems experience report follows my engineering work from July to October 2026 across coding clients, model-specific serving recipes, vLLM, SGLang, llama.cpp, and MLX. Models behind a nomina...

Vaibhav Jain · 0 citations
Sep 2026

Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests

AI coding agents now author a large share of pull requests (PRs) merged into popular open-source projects. A merged agent PR is usually considered finished work; yet, prior studies have reported issues in agent code after the merge (e.g., code smells and static-analysis issues). However, little is known about how often...

Wannita Takerngsaksiri, Nhat Duong, Scott Barnett · 0 citations
#artificial intelligence Review Oct 2026

Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

Coding agents can return plausible patches that omit required behavior. These failures are hard to review because long traces and confident summaries often hide what was missed. We ask when a nominally weaker reviewer can reliably decide whether a patch solves its issue. We study 411 execution-labeled traces from three...

Jun-Yu Guo, Shangding Gu, Ming Jin et al. · 1 citation
Preprint Oct 2026

Documentation Drift in AI-Generated Pull Requests

AI coding agents now contribute to software repositories at scale, yet their documentation practices remain uncharacterised. We analyse 29,669 AI-generated pull requests from 2,585 repositories in the AIDev dataset, measuring documentation co-change: whether code-file modifications are accompanied by a documentation-fi...

William DeMichele, Kla Tantithamthavorn · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.