Aug 2026· Applied Sciences· Vol 16, pp. 7887· 0 citations· 13 references
TL;DR
A Large Language Model-enhanced Autonomous Reinforcement Learning Penetration Testing framework that leverages the domain knowledge embedded in a Large Language Model to perform tactical planning, thereby pruning the original action space into a compact set of candidate actions.
Abstract
In recent years, Deep Reinforcement Learning (DRL) has emerged as a promising approach for automating penetration testing due to its capability to perform sequential decision-making in complex environments. However, in real-world enterprise networks, attack actions are typically characterized by highly coupled multi-dimensional parameter combinations, resulting in an exponentially expanding discrete action space. Such a large action space significantly degrades exploration efficiency and prevents conventional DRL agents from learning effective attack paths under sparse-reward conditions. To address these challenges, this paper proposes a Large Language Model-enhanced Autonomous Reinforcement Learning Penetration Testing framework (L-ARLPT). Specifically, the framework leverages the domain knowledge embedded in a Large Language Model (LLM) to perform tactical planning, thereby pruning the original action space into a compact set of candidate actions. Subsequently, an experience-driven layer employs the optimization mechanism of a Deep Q-Network (DQN) to conduct value estimation and policy learning within the reduced candidate set. To validate the effectiveness of the proposed framework, a high-fidelity enterprise penetration-testing simulation environment was constructed based on realistic enterprise attack scenarios. Experimental results demonstrate that, in a high-fidelity enterprise penetration-testing environment with a raw theoretical parameter-combination space containing 6×107 combinations, the proposed L-ARLPT framework achieves an average penetration depth of 3.33 out of 4.00, substantially outperforming both reinforcement learning baselines (all ≤1.27) and LLM-based baselines (1.12). Moreover, successful episodes require only 128.43 decision steps on average, enabling long-horizon, cross-domain lateral penetration in high-dimensional discrete action spaces.
In the quantitative finance area, particularly in order execution, reinforcement learning (RL) has shown great promise due to its ability to interact with market environments based on real data. However, traditional RL methods suffer from slow research speed and rely on static market assumptions, which do not consider the impact of the agent's execution action on the environment. To address these, we propose a Self-Evolutional single-agent/multi-agent Reinforcement Learning (SE-RL) framework. The framework utilizes a Large Language Model (LLM) to design various RL algorithm modules, such as agent model design, reward function, profiling, communication, and state imagination, by leveraging the LLM generating module output or code. SE-RL could continuously improve the accuracy of LLM-generated RL algorithms through a dual-enhancement kit at both high-level (prompt refinement) and low-level (parameter fine-tuning). Additionally, we use a multi-agent system to simulate dynamic financial markets, accounting for the impact of order executions on market dynamics. To further enhance training in such a dynamic market, we develop a hybrid environment training method that could rebalance each environment's loss weight. Comprehensive experiments on 200 realistic stock datasets demonstrate that our proposed framework outperforms current state-of-the-art baselines. Project page: https://kdd2026-se-rl.github.io/.
Vincent Fu, Xinxin Xu, Weichen Xu et al.· Proceedings of the 32nd ACM...· 0 citations
Despite recent advances in applying artificial intelligence (AI) techniques to radio access network (RAN), critical challenges remain: traditional machine learning (ML) algorithms suffer from limited generalization across varying network topologies, whereas general-purpose large language models (LLMs) face high computational demands and lack domain-specific knowledge. To address these gaps, this article introduces the evolving RAN intelligent controller (RIC) (EvoRIC) framework, a hierarchical architecture that enables continuous evolution by leveraging a non-real-time RIC (non-RT RIC) for global model updates and a near-real-time RIC (near-RT RIC) for local execution, dynamically empowering LLMs with domain-specific decision-making capabilities. Within this framework, we employ a reinforcement learning-based fine-tuning (RLFT) mechanism where an LLM operates as an actor within a proximal policy optimization (PPO) agent. By leveraging the interaction tuples collected from the wireless environment, the LLM's parameters are iteratively updated to align semantic reasoning with rigorous network performance objectives. We evaluate the generalization and efficacy of the proposed EvoRIC framework within integrated access and backhaul (IAB) networks, and finally, discuss the open challenges and future directions of the EvoRIC framework toward realizing autonomous O-RAN.
Lingyan Bao, Jemin Lee, Tony Q. S. Quek· 0 citations
Deep Reinforcement Learning (DRL) has achieved significant success in complex decision-making problems. As DRL systems are increasingly deployed in real-world applications, ensuring their quality and reliability is paramount. Current works primarily focus on detecting safety-critical failures, often neglecting policy optimality, which can lead to reduced efficiency, user distrust, and economic losses. This oversight, compounded by the inherent"testing oracle problem"for optimality, leaves a significant gap in comprehensively evaluating DRL systems. To address this gap, we propose Delta (Differential Testing for DRL Agents), a novel and comprehensive framework that automatically identifies both safety-critical and optimality bugs in DRL agents. Delta employs a two-phase approach: (1) Safety Testing, where the Agent Under Test (AUT) is evaluated for catastrophic failures while collecting data from its decision-making policy, and (2) Optimality Testing, where this collected data from the prior phase is used to train a challenger agent via Offline Reinforcement Learning. Differential testing is then performed by comparing the challenger agent against the AUT; instances where the challenger achieves higher cumulative rewards indicate optimality issues in the AUT. We demonstrate Delta's effectiveness across five environments. We investigate the effectiveness of three offline RL algorithms (BC, BCQ, and CQL) in generating challenger agents. Experimental results demonstrate that safety testing datasets are valuable for training competent DRL agents. Challenger agents trained with BCQ proved most effective for identifying optimality issues within the framework of Delta. Across the five environments, Delta uncovered an average of 2,518 optimality issues, outperforming the baseline methods by 50.2%.
Junda He, Jieke Shi, Zhou Yang et al.· 0 citations
High-dimensional state and action spaces com- bined with sparse reward structures in reinforcement learning (RL) environments typically require advanced control architec- tures. Hierarchical Reinforcement Learning (HRL) demonstrates superior performance compared to atomic RL approaches in these challenging scenarios. HRL can manage the complexity of commands to achieve task objectives through its hierarchical structure. One of the key challenges in HRL is efficiently training each level’s policy with optimal data collection from its experience. Off-policy correction is a critical technique for facilitating sample-efficient off-policy training in HRL, as it addresses the non-stationary issue of higher-level policy training. However, existing methods typically employ indirect probabilistic approaches that fail to accurately capture the current capability of the lower-level policy. This mismatch ultimately constrains the effectiveness of higher-level policy training. In this paper, we propose a novel HRL model that supports direct off-policy correction based on a Flow-based Deep Generative Model (FDGM). This approach leverages the inverse operation of FDGM to achieve goals aligned with the current knowledge of the lower-level policy. Additionally, our model addresses the limitations of FDGM to enable its effective use in HRL. Through comparative experiments on benchmark environments, our model demonstrates superior performance over existing models
Jaeyoon Kim, Junyu Xuan, C. Liang et al.· Journal of Artificial Intell...· 0 citations
The integration of Large Language Models (LLMs) with Reinforcement Learning (RL) for financial decision-making has grown rapidly in recent years, yet the literature remains fragmented and lacks systematic comparison across methods. In this survey we analyze 34 core studies (2023–2026), selected through a multi-stage process involving 84 initial candidates and 46 full-text reviews, and propose a three-paradigm taxonomy (feature-based, auxiliary-based, and policy-based) based on the functional role of LLMs within the RL pipeline. Analysis of these integration paradigms reveals an emergent architectural trade-off: while tighter policy-based coupling theoretically offers deeper contextual reasoning, it frequently introduces significant computational overhead and training instability. Conversely, simpler feature-based integration provides superior scalability and stability, though often at the expense of representational depth. Given the current benchmark fragmentation, the reported performance gains across these studies remain difficult to validate universally across different asset classes. Critical gaps identified include the insufficient handling of data leakage and look-ahead bias, standardized benchmarks, and limited alignment with regulatory frameworks such as MiFID II and the EU AI Act.
Ghusoon Hadi al-Aldaffaie, Alireza Taheri, Amirfarhad Farhadi et al.· Discover Artificial Intellig...· 0 citations
Recently, Reinforcement Learning (RL) has emerged as a crucial paradigm for the post-training of Large Language Model (LLM) agents. However, existing methods predominantly rely on sparse task rewards for policy optimization, failing to fully exploit another class of inherently dense supervisory signals naturally present during online interaction: environmental feedback following action execution. Recent theoretical studies suggest that generalization in multi-step, goal-oriented tasks hinges on predictive knowledge of environmental consequences. Inspired by this, we propose TAPO: Transition-Aware Policy Optimization for LLM Agents, a unified training framework that alternates between policy optimization and transition supervision. Beyond standard RL updates, TAPO repurposes rollout data to apply action-conditioned next-observation prediction supervision on a shared backbone model. This approach enhances the model's sensitivity to environmental transition dynamics and action consequences while concurrently optimizing the policy. It serves as a computationally lightweight, plug-and-play enhancement module for existing agent RL algorithms, requiring no additional expert data, extra sampling costs, or inference-time overhead. We conduct systematic experiments on WebShop and ALFWorld, integrating foundation models of various scales with different policy optimization algorithms. Empirical results demonstrate that TAPO consistently improves task performance over pure policy optimization baselines.
Cong Li, Peixi Peng, Yisen Zhao et al.· 0 citations