Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping

SafePG: Safe and Globally Optimal Reinforcement Learning with Hard Constraints

We present an optimal and convergent model-free policy gradient (PG) reinforcement learning (RL) framework for controlling nonlinear dynamical systems under hard safety constraints. We first construct a class of stochastic wrapper policies centered around a deterministic controller, thereby enabling exploration in unknown environments while preserving the underlying deterministic control structure. We then define a class of parameterized safe-by-construction control policies by truncating these stochastic policies onto hard safety constraints. We next establish, via measure-theoretic arguments, that the potentially nonconvex RL objective under the truncated policy class, as well as its policy gradients, are well-defined. We then develop a model-free PG algorithm based on stochastic gradient ascent that directly searches over these truncated policies and leverage gradient dominance to establish convergence and optimality guarantees. Finally, we validate this framework through simulations on a safe quadrotor navigation problem.

Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition

A unified framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA) is introduced that enables multiple actors to share a centralized multi-head critic and substantially improves both sample efficiency and policy performance.

GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control

Actor-critic architecture has been widely used in continuous robot control. However, they rely on learning a value network, introducing additional computational overhead during training. Moreover, policy learning may also be affected by the approximation error of value estimation. Critic-free group relative policy optimization methods provide a simpler training approach by removing the need for a critic. However, they fail to learn long-term action outcomes when directly applying immediate rewards to policy optimization in dense-reward environments. To address these problems, we propose Group Relative Return Policy Optimization (GR2PO), a critic-free reinforcement learning framework for continuous robot control. GR2PO estimates the discounted returns from the parallelly collected trajectories, performs group normalization at each rollout time index, and uses relative advantages and clipped targets to update the policy. To evaluate the effectiveness of the proposed framework, we instantiate it on robot control simulation environments and deploy the model to a real-world edge device. The results show that GR2PO significantly outperforms critic-free baselines that use immediate rewards and performs competitively against state-of-the-art actor-critic methods. Furthermore, GR2PO demonstrates competitive training efficiency. Inference tests on NVIDIA Jetson TX2 demonstrate the feasibility of deploying the learned policies on edge platforms. Further ablation experiments analyze the effects of parallel group size, return estimation methods, and target clipping ratio on learning performance. To support follow-up research, we will make the complete code publicly available after the paper is accepted, including the framework implementation, experimental configuration, and training and evaluation scripts.

Underactuated AUV course stability control based on optimal deep reinforcement learning with external disturbances and input constraints

Hierarchical reinforcement learning for safe output tracking: managing constraints of dependent internal states in unknown discrete-time systems

Reinforcement Learning for Real-Time Control Using Quanser Platforms: A Structured Narrative Review

Reinforcement learning (RL) is increasingly used for real-time control of complex dynamical systems, but its practical performance must be evaluated under hardware constraints that are often simplified in simulation. This paper presents a comprehensive review of published RL-based control studies using the Quanser Aero, Aero 2, 3-DOF Helicopter, and Autonomous Vehicles Research Studio (AVRS) platforms. The reviewed studies are compared according to the RL algorithm, control objective, hardware configuration, implementation environment, and reported experimental performance. The synthesis shows that Aero and Aero 2 are used primarily for stabilisation, trajectory tracking, and energy-aware control, whereas the 3-DOF Helicopter provides a more demanding benchmark for adaptive and Actor–Critic methods under nonlinear and coupled dynamics. AVRS offers significant potential for vision-based and multi-agent RL; however, the available experimental literature remains limited. Across the reviewed comparisons, policy-gradient and Actor–Critic methods, including PPO and SAC, generally demonstrate greater adaptability and smoother continuous-control behaviour, while conventional controllers frequently retain advantages in steady-state accuracy, computational predictability, and safety verification. Nevertheless, no RL algorithm can be identified as universally superior because the published studies employ heterogeneous reward functions, reference trajectories, sampling rates, performance measures, and hardware configurations. Recurring limitations include sample inefficiency, simulation-to-hardware discrepancies, computational latency, safety constraints, and incomplete reporting of experimental protocols. The review therefore identifies standardised evaluation procedures, reproducible reporting, safety-aware RL, interoperable software interfaces, and higher-fidelity digital twins as priorities for future research. No new experimental data are generated; the contribution is a comparative synthesis of experimental evidence reported in the literature.