Reinforcement Learning Model Predictive Control for Long-Term Regulation

Abstract

Liquid-level control in coupled tank systems poses challenges due to their nonlinearity and multiple time scales. Solving this problem is crucial for quality control, production optimization, and process flexibility in industries such as oil refining, water treatment, and chemical mixing. The use of coupled tanks is an effective way to simplify a multi-time system, as it captures both short- and long-term impacts within this nonlinear framework. As a solution, the model predictive control (MPC) offers a viable approach for short-horizon control, while model-free reinforcement learning (RL) methods have proven effective for addressing long-term effects in control systems. Two approaches to integrate MPC and RL, leveraging their advantages for short-term path tracking while minimizing long-term effects in coupled tank systems, are presented. Experimental findings indicate that the first method, adjusting the MPC setpoint, is beneficial in scenarios where long-term minimization is crucial, reducing the tank 2 level by 10.4% compared to short-horizon MPC. The second method, which integrates an RL control action into the MPC output, is advantageous when this new sudden change in the output is feasible for the actuator. With 3.5 times greater maximum control effort variation, steady-state error is reduced by 61.3% compared to short-horizon MPC. Both methods effectively address unmodeled dynamics and reduce the final steady-state error.