Hybrid Computed Torque Control and Soft Actor-Critic Framework for Sample-Efficient Robotic Manipulator Control

Abstract

Reinforcement Learning (RL) has emerged as a promising approach for robotic control, enabling agents to learn control policies through interaction with complex and dynamic environments. However, standalone RL methods often suffer from poor sample efficiency, limiting their practicality for real-world robotic systems. To address this limitation, recent studies have combined RL with classical controllers such as proportional–integral–derivative (PID) control, where the classical controller provides a baseline policy and RL learns residual corrective actions. Nevertheless, conventional PID controllers do not explicitly incorporate the full nonlinear manipulator dynamics.This paper proposes a physics-informed residual reinforcement learning framework that combines Computed Torque Control (CTC) with Soft Actor-Critic (SAC) for trajectory tracking of a 2-DOF robotic manipulator. The CTC component utilises analytical Lagrangian dynamics to provide a nominal control torque, while SAC learns bounded residual corrections to compensate for model uncertainties and unmodelled effects. The proposed framework is evaluated in CoppeliaSim and compared against CTC-only, RL-only, and PID+SAC baselines under identical experimental conditions.Experimental results demonstrate that the proposed CTC+SAC framework achieves the lowest mean and steady-state tracking errors among all evaluated methods, with a 4.9% reduction in mean error over RL-only and a 3.3% reduction over PID+SAC within the considered simulation setup. The results suggest that incorporating analytical robot dynamics into the residual learning framework improves tracking performance and sample efficiency compared to both pure RL and classical controller baselines.