This work presents an end-to-end diagnostic study of an Autonomous Adversary system with orchestrator, executor, and validator LLMs in enterprise-like lateral-movement scenarios and uses comparative LLM-as-a-Judge analysis to identify planning deficiencies.
Abstract
Multi-stage LLM-based cyber agents may complete attack workflows while remaining brittle, costly, or reliant on incorrect interpretations of execution evidence. Success rates alone obscure inefficiency, adaptation through retries, and recognition of success or failure. We present an end-to-end diagnostic study of an Autonomous Adversary system with orchestrator, executor, and validator LLMs in enterprise-like lateral-movement scenarios. Six frontier models are evaluated across two scenarios and three modes: expert-defined, self-scaffolded, and fully autonomous. We assess validator consistency and evidence grounding; introduce a subtask-conditioned, cost-aware score for abnormal token use, retries, and runtime; and use comparative LLM-as-a-Judge analysis to identify planning deficiencies, including tool misalignment, plan similarity, over-specification, inadequate probing, and weak recovery. Validators are generally relevant and evidence-grounded but often nonspecific and overly optimistic. Bottlenecks cluster in credential and lateral-movement tasks, spread with scenario complexity, and vary more under full autonomy. Reliable evaluation must assess outcomes, evidence interpretation, resource use, and adaptation after failure.
This work argues that agentic risk is progressive: it can enter at four loci of the agent control loop--skill admission, invocation-time intent, execution-time effect, and post-action consequence--while a denied dangerous objective can reappear across surface forms, tools, or turns.
Kai Wang, Zeming Wei, Biaojie Zeng et al.· 0 citations
RedAgentBench is introduced, an executable framework for autonomous red-teaming and faithful measurement that shows that executable evaluation can improve safety measurement and identify actionable intervention points.
Zixing Chen, Xingyuan Liu, Jie Zhu et al.· 3 citations
A four-dimensional Integration Friction Index is introduced that separates one-time engineering cost from recurring organisational, legal, and maintenance cost and shows why scope and budget enforcement cannot be delegated to system prompts.
Israt Moyeen Noumi, Tarannum Ahmed Nowshin, Md Mehedi Hasan Nipu et al.· 0 citations
MazeRunner is presented, an autonomous penetration testing system built on a three-agent task-and-clue orchestration framework that separates global orchestration, context-intensive execution, and failure-oriented review while persistently maintaining task states and environmental evidence.
Zhenyuan Li, Yi Jiang, Junjie Cheng et al.· 0 citations
This study introduces SymTrace, a controlled evaluation framework that records the MAS execution trajectory and establishes intervention anchors and explores the effectiveness of MAS repair methods, revealing that existing unguided rerun methods are highly unreliable.
Zhong-Wen Luan, Xiaoyan Zhang, Ming Hu et al.· 2 citations
This work systematizes MAS security through an execution-centered analysis of 197 works, introducing an A-I-R framework that organizes attacks by adversary position, interaction interface, and resulting system-level risk, unifying otherwise fragmented attack mechanisms across MAS.
Rui Yang, Jun-Jie Xu, Zheng-Yu Liu et al.· 3 citations