Autonomous AI agents are increasingly being used to assist software developers in tasks such as code generation and refactoring. However, we still lack a clear understanding of how these agents interact with humans in collaborative workflows such as GitHub pull requests (PRs). A helping hand to study this phenomenon comes from the AIDev dataset featuring 932k PRs authored (i.e., opened) by AI agents across 116k GitHub repositories. Nevertheless, such a dataset provides a limited view of the collaborations between humans and agents, since the latter may also be involved in PRs opened by humans. In this work, we analyse interaction patterns between humans and AI agents in PRs. To support this analysis, we introduce a novel methodology that extends the AIDev dataset to capture all PRs involving agent participation, irrespective of whether the agents initiated the PR. We apply such a methodology to a subset of 383 repositories from AIDev, finding 6,445 PRs, out of which 5,118 are agentrelevant (vs the 3,197 of AIDev on the same repositories) for a total of 74,525 interactions from 671 unique users. We classify these PRs by authorship (i.e., agent, bot, or human) and examine which participant types perform key interactions (e.g., reviews). Our findings indicate that human-agent collaboration is the most common interaction pattern, with about nine out of ten changes being merged. Although agents author a substantial proportion of PRs, fully autonomous contributions remain uncommon and are rarely accepted. These results highlight the increasing importance of AI agents in augmenting human capabilities in software development, as well as the central role of human oversight in ensuring successful contributions.
Miguel Romero-Arjona, Saman Barakat, Alberto Martin-Lopez et al.· Annual International Compute...· 0 citations
Large Language Models (LLMs) unlocked new possibilities in automated code writing, becoming the backbone of most code completion tools. While LLMs excel in mainstream languages, they often lack support for the so-called low-resource languages where training data is scarce. As a result, these languages lag behind in the quality of code completion tooling available to their communities. A concrete example is Pharo, a Smalltalk-inspired language whose IDE currently offers only single-token completion. In this work, we report on our experience bringing LLM-based code completion to Pharo. First, we describe an end-to-end pipeline that combines Pharo-specific data curation, continued pre-training and fine-tuning of open code LLMs. Second, we introduce a set of Pharo code completion benchmarks designed to evaluate whether models (i) learn Pharo's syntax and (ii) accurately complete masked Pharo code from real-world GitHub repositories. Third, we show empirically that Pharo-specialized models substantially outperform their original base checkpoints and also exceed the accuracy of substantially larger code LLMs on Pharo completion. Overall, our case study demonstrates the feasibility of bringing strong LLM-based code completion to low-resource programming languages, with models small enough to provide ``real-time''in-IDE support.
K. Kier, Alessandro Giagnorio, Omar AbedelKader et al.· 1 citation