When AI Joins the Team: Understanding Human-Agent Collaboration in Pull Requests

Abstract

Autonomous AI agents are increasingly being used to assist software developers in tasks such as code generation and refactoring. However, we still lack a clear understanding of how these agents interact with humans in collaborative workflows such as GitHub pull requests (PRs). A helping hand to study this phenomenon comes from the AIDev dataset featuring 932k PRs authored (i.e., opened) by AI agents across 116k GitHub repositories. Nevertheless, such a dataset provides a limited view of the collaborations between humans and agents, since the latter may also be involved in PRs opened by humans. In this work, we analyse interaction patterns between humans and AI agents in PRs. To support this analysis, we introduce a novel methodology that extends the AIDev dataset to capture all PRs involving agent participation, irrespective of whether the agents initiated the PR. We apply such a methodology to a subset of 383 repositories from AIDev, finding 6,445 PRs, out of which 5,118 are agentrelevant (vs the 3,197 of AIDev on the same repositories) for a total of 74,525 interactions from 671 unique users. We classify these PRs by authorship (i.e., agent, bot, or human) and examine which participant types perform key interactions (e.g., reviews). Our findings indicate that human-agent collaboration is the most common interaction pattern, with about nine out of ten changes being merged. Although agents author a substantial proportion of PRs, fully autonomous contributions remain uncommon and are rarely accepted. These results highlight the increasing importance of AI agents in augmenting human capabilities in software development, as well as the central role of human oversight in ensuring successful contributions.