Agentic AI for Offensive Security: LLM-guided Autonomous Red Teaming in a Limited Cyber-range Environment

Abstract

Agentic artificial intelligence (AI) is increasingly being explored for automating offensive security and red teaming tasks, enabling systems that can coordinate multi-step cyber operations through structured decision-making. While prior research has investigated reinforcement learning (RL) agents and large language models (LLMs) for penetration testing, most studies are evaluated in simulated or abstract environments, with limited empirical validation in real cyber-range settings. This paper presents a controlled experimental evaluation of an LLM-guided offensive security pipeline against deterministic scripted baselines in a cyber-range environment. Using a vulnerable Kioptrix virtual machine and a Kali Linux attacker, we implement three deterministic pipelines, fixed-path, sequential and rule-based, alongside two configurations of an LLM-guided agent: an initial version (LLM V1) and a refined constrained controller (LLM V2). All approaches operate within a restricted and auditable action space executed through predefined tools. Across repeated trials, the initial LLM configuration exhibits reduced reliability and increased execution cost due to exploratory behaviour. In contrast, the refined controller achieves a 100% success rate, reduces execution steps, eliminates wasted actions and matches the efficiency of rule-based automation. These results show that, within a controlled cyber-range environment, LLM-guided agents can approximate deterministic performance when appropriate constraints are applied. This suggests that agent-based approaches to offensive security can support semi-autonomous red teaming workflows, provided that decision-making is governed by structured control policies.