A ScholForge - hierarchical multi-agent LLM system that tackles end-to-end software engineering research end-to-end, with a natural-language research goal, and produces working code, experimental results, and a compiled manuscript without human intervention.
Abstract
Software engineering research spans many activities: surveying literature, building software, running experiments, writing papers-that typically require different tools and expertise at each stage. We present a ScholForge - hierarchical multi-agent LLM system that tackles this end-to-end. Manager agent delegates to five specialists (Librarian, Proposer, MLE Solver, Writer, Reviewer) through a Code-Act execution loop, with external capabilities exposed via three Model Context Protocol (MCP) servers. Given a natural-language research goal, the system produces working code, experimental results, and a compiled manuscript without human intervention. We present one representative case study: Prompt2Edge, an autonomously developed agentic ML pipeline for edge devices that achieved 86.71% test accuracy on Speech Commands with a 17.8k-parameter model and 0.9638 Macro-F1 on UCI HAR, consuming approximately 5 M tokens and completing the full pipeline in under 5 hours.
Research on integrating specialized, language-model-based agents for automated test-case generation remains limited, and the detection criteria under which such systems are evaluated have not been standardized, which makes reported rates difficult to compare across studies. This paper presents a multi-agent testing framework in which requirement analysis, test-case generation, sandboxed execution, and defect detection are delegated to four distinct agents, and in which detection rests on a composite rule combining the execution signal with the semantic judgment of a dedicated diagnostic agent. The framework is evaluated on the QuixBugs dataset of forty Python programs under three criteria of increasing strictness. A two-run differential protocol against the reference implementation, recommended here as the primary indicator, yields a detection rate of 55.0%, against 90.0% under the unconditional criterion conventional in prior work and 5.0% under a strict criterion requiring the suite to pass entirely on correct code. The thirty-five-point gap is traced to language-model test artifacts, of which hallucinated oracle values and signature mismatches account for the majority. An ablation with a same-model single-agent baseline attributes the framework’s advantage to role decomposition rather than to the diagnostic agent specifically, and the distribution of detection across the fourteen defect classes defined by the benchmark is reported descriptively.
Yuxuan Li· Transactions on Computing Sc...· 0 citations
Automating software development is an important aspect of software engineering. Generative Artificial Intelligence (GenAI) Agents that control multiple tools and use feedback with some autonomy to iteratively solve problems are emerging as promising tools for increased software production. To achieve autonomous software maintenance with minimal human involvement, such agents must not only solve human-posed problems but also decide which problems to address. We report our experiences developing a GenAI agent that autonomously formulates, prioritizes, and completes a limited set of code improvement tasks enterprise-wide, and we evaluate this agent from two perspectives: first, developing a methodology to estimate effort savings from such autonomous agents by comparing time spent on agent-introduced tasks and agent maintenance efforts against manual execution; and second, comparing the impact of manual versus agent-driven code improvements on future codebase maintenance. Our results demonstrate significant effort savings, even after accounting for effort in agent development, and a Difference-in-Differences (DiD) analysis found that automated improvements had an effect on the reduction of future maintenance effort that was as strong as or stronger than manual improvements confirming the feasibility of autonomous maintenance. While the relaxed completion and timeliness needs of perfective maintenance make it an ideal candidate for autonomy, a degree of autonomy may be achievable in other areas of software development as well.
A comprehensive overview of the existing tools and frameworks for implementing MAS in software engineering and a set of lessons learned and challenges that can help researchers and practitioners to select a suitable MAS framework according to their needs are provided.
Mariama Celi Serafim De Oliveira, M. Ibiyo, Marco Gianrusso et al.· 0 citations
Agent Gym is introduced, a modular, domain-agnostic framework that wraps any existing LLM-based agent in a continuous evaluation-and-evolution loop and introduces the Spec-to-Note Gap, an autoencoder-inspired view of agentic system transparency.
Pouya Ghiasnezhad Omran, Michael Zimmermann, Duncan Cambridge et al.· 0 citations
Multi-agent systems built on large language models (LLMs) are increasingly deployed for complex tasks requiring autonomous planning, tool use, and inter-agent coordination. However, the non-deterministic nature of LLM outputs and the emergent behavior arising from agent interactions render traditional test oracles ineffective, creating a critical gap in quality assurance for agentic AI. This work introduces MORPHAGENT, a framework designed to address the oracle problem in multi-agent LLM systems through trace-based behavioral analysis. Our contributions are threefold: (1) goal-preservation relations that verify consistent goal achievement under input perturbations, (2) coordination-consistency relations that validate inter-agent delegation and communication patterns under agent substitution and reordering, and (3) tool-use integrity relations that ensure semantic equivalence of tool invocation sequences under prompt paraphrasing. MorphAgent instruments agent execution to capture structured traces comprising planning steps, tool calls, message exchanges, and final outputs, then systematically applies metamorphic transformations and checks behavioral invariants without requiring ground-truth oracles. We evaluate the framework on four multi-agent benchmarks spanning code generation, research synthesis, customer service, and data analysis tasks, encompassing 2,840 source-followup execution pairs across three LLM backends. Results show that MORPHAGENT detects 82.0% of seeded behavioral faults, including 90.3% of coordination failures and 81.7% of goal-deviation faults, while maintaining a false positive rate of 6.1%. The framework uncovers 14 previously unreported behavioral anomalies in established multi-agent frameworks, demonstrating its practical utility for assuring agentic AI reliability. These results suggest that trace-based metamorphic testing can serve as a practical foundation for reliable validation of emerging agentic AI systems.
Gopalakrishnan Marimuthu· International Conference on...· 0 citations
AutoMOOSE is an agentic software framework that converts a natural-language request into an executed, screened, and interpreted MOOSE phase-field simulation. Here, we deploy AutoMOOSE as a agentic software, complementing our prior work which focused on development of the agentic tool. We describe our software framework and architecture through Use Case and logical views of the 1+5 architectural-views model, covering its user roles, component structure, six-agent pipeline, physics plugin layer, Model Context Protocol interface, and screening/falsification/recovery loop. Our architecture separates physical falsification from automatic repair, so corrected simulations remain inspectable and must be re-admitted before acceptance. We focus on software design, extensibility, interoperability, and reuse of the AutoMoose framework for broad utilization in multiphysics materials design problems.
Sukriti Manna, Henry Chan, Subramanian K. R. S. Sankaranarayanan· 0 citations