Jul 2026· Annual International Computer Software and Applications Conference· pp. 537-542· 0 citations· 15 references
Abstract
Refactoring is essential for maintaining and evolving software systems, yet we still have limited insight into how automated code-generation agents describe these changes in pull requests. In this study, we analyze refactoring-related pull requests produced by five AI coding agents, with a particular focus on how their intentions are communicated through pull request descriptions. To better understand recurring transformation patterns, we extract refactoring descriptors terms that capture different refactoring activities and use them to identify similarities across agents. We then classify the pull requests into three main categories: internal quality attributes, external quality attributes, and code smells. Our dataset includes 2,288 unique AIgenerated refactoring pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. The results reveal several limitations in how AI systems perform and document refactoring, highlighting the need for clearer and more structured communication in collaborative software development.
Refactoring is essential for maintaining and evolving software systems, yet we still have limited insight into how automated code-generation agents describe these changes in pull requests. In this study, we analyze refactoring-related pull requests produced by five AI coding agents, with a particular focus on how their intentions are communicated through pull request descriptions. To better understand recurring transformation patterns, we extract refactoring descriptors terms that capture different refactoring activities and use them to identify similarities across agents. We then classify the pull requests into three main categories: internal quality attributes, external quality attributes, and code smells. Our dataset includes 2,288 unique AIgenerated refactoring pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code. The results reveal several limitations in how AI systems perform and document refactoring, highlighting the need for clearer and more structured communication in collaborative software development.
Aymen Masmoudi, Belhassen Khefacha, Andrew Haralambous et al.· Annual International Compute...· 0 citations
Code refactoring aims to enhance the internal structure of source code without affecting its functional behavior. The recent advancements of Large Language Models (LLMs) have demonstrated potential for automating software engineering tasks, such as code refactoring. However, the refactorings produced by LLMs often introduce subtle errors, leading to functional behavior changes and failed unit tests, which limit their practical adoption. To address the limitations of LLM-generated refactorings, we analyze the root causes of their failures and develop the RefactorAssist agent to improve the functional correctness of LLM-generated refactorings. To this end, we use 10 open-source Java projects with their native test suites and manually evaluate why LLM-generated refactorings fail unit tests. We then design an agentic approach that leverages unit-test logs, error explanations, project context retrieval, and code diffs to guide the iterative refactoring. Our findings show that the main reasons for failure are context misunderstanding/hallucination (24.3%), incorrect or inconsistent renaming (15.3%), adding new functionality or variables (13.7%), code incompleteness (11.3%), syntax and structural errors (9.7%), edge cases not handled (9%), improper type handling (8.7%), and variables outside scope (8%). To make our approach cost-effective, RefactorAssist first applies a static repair step for missing imports, unbalanced brackets, and compilation errors without LLMs. For remaining failures, RefactorAssist incorporates error logs and code diffs, achieving up to a 70.8% repair rate on the remaining failures and a 94.2% cumulative pass rate under the best-performing configuration. These results indicate that static checks and test-guided, context-aware agentic repair can increase the reliability of LLM-generated refactorings, bringing them closer to practical integration within developer workflows.
Jonathan Cordeiro, Shayan Noei, Ying Zou· 0 citations
This paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face, and describes a seven-stage workflow and five process shifts, including a move toward evaluation-driven development.
Yunbo Lyu, David Williams, Jieke Shi et al.· 0 citations
This study characterize agentic pull requests (PR) in comparison to human generated PRs and to examine how their properties change across different stages of the development lifecycle, offering a more nuanced understanding of their benefits and limitations in real-world practices.
Iren Mazloomzadeh, Mohammad Mehdi Morovati, F. Khomh· 0 citations
AI agents are becoming a fundamental part of modern software creation, helping developers in generating code, debugging, designing systems, etc. But there is a clear difference between how beginners and experienced software engineers get benefits from these tools. Newbies usually depend on agents for one-time prompts and quick answers, whereas mature users utilize them through well-defined, repeated workflows that raise productivity and consistency. In this article, we discuss this difference and emphasize that getting the full potential does not merely depend on better prompts but on workflows driven by instructions developers create clear and reusable instruction files to direct agent behavior across tasks. When developers stop seeing agents only as chat interfaces but as programmable collaborators, they can produce more reliable and high-quality outputs. We offer in our paper methods like designing modular instructions, narrowing down the context, and iterative refinement loops, as well as a case study illustrating how a team made a code review more efficient and minimized the rework by making agent instructions standard. The results stress that structured forms of interaction rather than sporadic use are the main ways to tap into advanced features. Our paper provides a conceptual model for agent usage at large scale, hands-on advice for the implementation of instruction files in actual settings, and validation that skillful developers can far exceed basic usage by adopting orderly, system-like approaches to agent collaboration.
Madhurima Kommuru, Srujana Pulipaka· International Journal of Mod...· 0 citations
Maintaining up-to-date code documentation is difficult in fast-moving repositories because design knowledge is scattered across source files and pull requests. We present CODENS, a system that turns pull requests into living, accessible, and queryable documentation for production codebases. CODENS incrementally builds a typed software knowledge graph from pull requests, enriches components through schema-driven semantic extraction, derives typed relations between them, and exposes the resulting knowledge through three retrieval modes, including agent-guided graph traversal for repository-level question answering. The system also preserves semantic change history across pull requests and integrates both answer-quality and operational evaluation metrics. We evaluate CODENS on a client Ruby on Rails project in production. Results show that CODENS produces highly relevant and well-grounded answers, while qualitative feedback highlights a remaining challenge in concise, documentation-oriented synthesis.
Abdelhak Kelious, Chyrine Tahri, Eliot Bardet· Proceedings of the 2026 ACM...· 0 citations