This study characterize agentic pull requests (PR) in comparison to human generated PRs and to examine how their properties change across different stages of the development lifecycle, offering a more nuanced understanding of their benefits and limitations in real-world practices.
Abstract
Recent advances in large language models and their rapid adoption across software engineering tasks have made Artificial Intelligence (AI) coding agents an integral component of modern software development workflows. While developers increasingly benefit from these coding agents, their impact on software quality remains insufficiently understood. In particular, how agentic contributions evolve across the software development lifecycle has not been thoroughly investigated. This study aims to characterize agentic pull requests (PR) in comparison to human generated PRs and to examine how their properties change across different stages of the development lifecycle. Using the AIDev dataset, we first analyze how differences in merge rates between agentic and human generated PRs vary over time. We then identify the types of development tasks where AI coding agents are predominantly applied and investigate how these task distributions evolve across development quarters. Finally, we compare a set of key characteristics of agentic and human generated PRs, focusing on their implications for software quality and their temporal dynamics. Overall, our findings provide an empirical and longitudinal perspective on the role of AI coding agents in software development, offering a more nuanced understanding of their benefits and limitations in real-world practices.
This paper empirically study the development and release evolution of five major open-source agent harnesses, revealing extreme release velocities exceeding two releases per day and thousands of issues within months, and performs the first controlled longitudinal study that isolates the agent harness contribution.
O. Sghaier, Hao Li, Bram Adams et al.· 2 citations
A three-level taxonomy inspired by autonomous driving that distinguishes degrees of autonomy along a roadmap from today’s AI-assisted development workflows to fully autonomous software development in which AI systems autonomously identify demands and design, implement, verify, and maintain software without human oversight is introduced.
Large-scale human–agent collaboration is investigated by comparing 40,214 pull requests across 2,807 GitHub repositories, including 33,596 agent-authored PRs from five autonomous coding agents and 6,618 human-authored PRs, revealing a socio-technical trade-off.
A. Njoku, Zohreh Sharafi, F. Khomh· AIware· 2 citations
Refactoring is essential for maintaining and evolving software systems, yet we still have limited insight into how automated code-generation agents describe these changes in pull requests. In this study, we analyze refactoring-related pull requests produced by five AI coding agents, with a particular focus on how their intentions are communicated through pull request descriptions. To better understand recurring transformation patterns, we extract refactoring descriptors terms that capture different refactoring activities and use them to identify similarities across agents. We then classify the pull requests into three main categories: internal quality attributes, external quality attributes, and code smells. Our dataset includes 2,288 unique AIgenerated refactoring pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. The results reveal several limitations in how AI systems perform and document refactoring, highlighting the need for clearer and more structured communication in collaborative software development.
Aymen Masmoudi, Belhassen Khefacha, Andrew Haralambous et al.· Annual International Compute...· 0 citations
Refactoring is essential for maintaining and evolving software systems, yet we still have limited insight into how automated code-generation agents describe these changes in pull requests. In this study, we analyze refactoring-related pull requests produced by five AI coding agents, with a particular focus on how their intentions are communicated through pull request descriptions. To better understand recurring transformation patterns, we extract refactoring descriptors terms that capture different refactoring activities and use them to identify similarities across agents. We then classify the pull requests into three main categories: internal quality attributes, external quality attributes, and code smells. Our dataset includes 2,288 unique AIgenerated refactoring pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. The results reveal several limitations in how AI systems perform and document refactoring, highlighting the need for clearer and more structured communication in collaborative software development.
Aymen Masmoudi, Belhassen Khefacha, Andrew Haralambous et al.· Annual International Compute...· 0 citations
This work uses event logs extracted from software repositories to discover project-specific agent roles using a predefined SE role vocabulary grounded in repository behavior and generates matching agent specifications and implementations that are aligned with human expectations.
Saimir Bala, Fabiana Fournier, Lior Limonad et al.· 0 citations