EVALUATING THE OFFENSIVE UTILITY OF LARGE LANGUAGE MODELS IN CYBER THREAT GENERATION

Abstract

Large Language Models (LLMs) are increasingly integrated into cybersecurity workflows, creating both defensive opportunities and new forms of offensive enablement. Existing research demonstrates capabilities in threat modeling, attack-path construction, Cyber Threat Intelligence (CTI) processing, adversarial text generation, and automated penetration testing; however, these capabilities are commonly evaluated in isolation. This review synthesizes recent evidence and develops a methodological framework for evaluating offensive utility as a multidimensional construct encompassing technical capability, correctness, contextual grounding, reliability, operational relevance, temporal efficiency, and human-expertise compression. The review distinguishes linguistic plausibility from cyber validity and separates model capability from system-level utility. Evidence from recent studies indicates that LLMs can support attack-technique mapping and multi-stage reasoning, while retrieval-augmented architectures can improve contextualization. At the same time, hallucination, inconsistent reasoning, latency, safety restrictions, and retrieval dependence constrain practical utility. The proposed framework therefore treats offensive utility as an emergent property of the interaction among model, context, retrieval, tools, and human oversight rather than as a single model attribute. The framework provides a basis for reproducible comparative evaluation without requiring unauthorized real-world offensive activity.