Disturbance-Rejection Reinforcement Learning for Control of Unknown Nonlinear Systems

Abstract

How to realize the ultimate boundedness during training on unknown nonlinear systems under continuous stochastic disturbances is a key challenge in reinforcement learning (RL). To this end, we propose a Lyapunov-based data-driven RL framework for disturbance rejection control and provides a step toward reliable disturbance-rejection learning. First, a stabilizable reference model is constructed via input-convex neural networks (ICNNs) to generate stabilizable, disturbance-aware trajectories from data that reduce sensitivity to nominal model mismatch. Second, we establish a data-driven ultimate-boundedness theorem. It certifies closed-loop stability directly from sampled transitions without requiring prior model knowledge. Building on these foundations, we propose an off-policy algorithm that alternates between reference-model learning and policy optimization. It enforces Lyapunov decrease in expectation and jointly improves performance and robustness to achieve ultimate boundedness. Experiments on a classic control task and MuJoCo benchmarks demonstrate that the proposed method reduces the final-20-episode average cost by up to 75% and achieves average episode lengths over the final 20 episodes up to 3.2 times longer than those of representative Lyapunov-based RL methods. Note to Practitioners—Nonlinear systems such as robotics and autonomous vehicles often encounter challenges of external disturbances and uncertain dynamics. To address these issues, this paper proposes a reinforcement learning (RL)-based control framework specifically designed for such environments. By integrating a stabilizable reference model and Lyapunov-based stability guarantees, the method ensures both robustness and control performance. The framework is particularly valuable in settings with common disturbances such as sensor noise or actuator errors. It enables policy optimization without requiring detailed system models, enhancing adaptability and scalability for high-dimensional nonlinear systems. Furthermore, the framework guarantees bounded state trajectories, ensuring safety guarantees while maintaining performance under unknown conditions.