Skip to content

Where Cyber Agents Struggle: Bottleneck Analysis of Multi-Stage LLM Agents

Sep 2026 · arXiv.org · Vol abs/2609.28572 · 0 citations · 7 references
Computer Science

TL;DR

This work presents an end-to-end diagnostic study of an Autonomous Adversary system with orchestrator, executor, and validator LLMs in enterprise-like lateral-movement scenarios and uses comparative LLM-as-a-Judge analysis to identify planning deficiencies.

Abstract

Multi-stage LLM-based cyber agents may complete attack workflows while remaining brittle, costly, or reliant on incorrect interpretations of execution evidence. Success rates alone obscure inefficiency, adaptation through retries, and recognition of success or failure. We present an end-to-end diagnostic study of an Autonomous Adversary system with orchestrator, executor, and validator LLMs in enterprise-like lateral-movement scenarios. Six frontier models are evaluated across two scenarios and three modes: expert-defined, self-scaffolded, and fully autonomous. We assess validator consistency and evidence grounding; introduce a subtask-conditioned, cost-aware score for abnormal token use, retries, and runtime; and use comparative LLM-as-a-Judge analysis to identify planning deficiencies, including tool misalignment, plan similarity, over-specification, inadequate probing, and weak recovery. Validators are generally relevant and evidence-grounded but often nonspecific and overly optimistic. Bottlenecks cluster in credential and lateral-movement tasks, spread with scenario complexity, and vary more under full autonomy. Reliable evaluation must assess outcomes, evidence interpretation, resource use, and adaptation after failure.

View source

Similar papers

Review Aug 2026

ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents

This work argues that agentic risk is progressive: it can enter at four loci of the agent control loop--skill admission, invocation-time intent, execution-time effect, and post-action consequence--while a denied dangerous objective can reappear across surface forms, tools, or turns.

Kai Wang, Zeming Wei, Biaojie Zeng et al. · 0 citations
Preprint Aug 2026

REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems

RedAgentBench is introduced, an executable framework for autonomous red-teaming and faithful measurement that shows that executable evaluation can improve safety measurement and identify actionable intervention points.

Zixing Chen, Xingyuan Liu, Jie Zhu et al. · 3 citations
Preprint Aug 2026

Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing

A four-dimensional Integration Friction Index is introduced that separates one-time engineering cost from recurring organisational, legal, and maintenance cost and shows why scope and budget enforcement cannot be delegated to system prompts.

Israt Moyeen Noumi, Tarannum Ahmed Nowshin, Md Mehedi Hasan Nipu et al. · 0 citations
Review Aug 2026

MazeRunner: Nonlinear Task and Clue Orchestration for LLM-driven Black-Box Automated Penetration Testing

MazeRunner is presented, an autonomous penetration testing system built on a three-agent task-and-clue orchestration framework that separates global orchestration, context-intensive execution, and failure-oriented review while persistently maintaining task states and environmental evidence.

Zhenyuan Li, Yi Jiang, Junjie Cheng et al. · 0 citations
Preprint Aug 2026

Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems

This study introduces SymTrace, a controlled evaluation framework that records the MAS execution trajectory and establishes intervention anchors and explores the effectiveness of MAS repair methods, revealing that existing unguided rerun methods are highly unreliable.

Zhong-Wen Luan, Xiaoyan Zhang, Ming Hu et al. · 2 citations
#artificial intelligence Preprint Sep 2026

SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems

This work systematizes MAS security through an execution-centered analysis of 197 works, introducing an A-I-R framework that organizes attacks by adversary position, interaction interface, and resulting system-level risk, unifying otherwise fragmented attack mechanisms across MAS.

Rui Yang, Jun-Jie Xu, Zheng-Yu Liu et al. · 3 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.