Beyond Outcomes: Toward Process-Aware Software Agents

Abstract

Autonomous programming agents are commonly evaluated by whether they solve software tasks, providing limited insight into the execution processes that lead to success or failure. We present a process-centric approach to evaluating agent behavior through trajectories. We introduce Graphectory, which captures the structural organization of agent execution, and Langutory, which abstracts low-level actions into software engineering phases to characterize problem-solving strategies. We further relate observed execution to intended workflows through plan compliance. Across thousands of trajectories, our analysis uncovers behavioral patterns that outcomes alone cannot explain, including persistent inefficiencies in successful runs and substantial variation in how agents translate plans into execution. These findings highlight the importance of understanding how agents execute, laying the foundation for the process-aware optimization of software agents.