Offline Reinforcement Learning Using Distributional Representation for Autonomous Uncrewed Surface Vehicles Navigation

Abstract

Uncrewed surface vehicles (USVs) have the potential to create ocean Internet of Things (IoT) in a dynamic maritime traffic environment. The autonomous navigation of USVs in complex unknown environments poses a core bottleneck for deploying such marine communication systems. In recent years, the deep reinforcement learning (DRL) algorithms have been utilized to learn intelligent navigation policies via a trial-and-error training paradigm. Nevertheless, traditional online DRL suffers prominent drawbacks when deployed at sea, e.g., high exploration risks, low sample efficiency, and slow convergence speed. These limitations severely restrict its practical deployment scope in real maritime environments. To address this issue, this article proposes an offline reinforcement learning algorithm equipped with distributional value functions for multi-USV navigation in congested water areas. First, a behavior cloning (BC)-based regularizer is employed into the loss function of actor network to address the distributional shift problem of offline learning. Second, the distributional representation of accumulative reward is approximated by critic network, which can help accelerate and stabilize the training process. Finally, extensive experiments have been conducted to test the proposed algorithm. The simulation results demonstrate that our algorithm can learn effective navigation policies comparable to those trained by online RL within much more time steps. Moreover, by decoupling data collection from policy optimization, the proposed framework eliminates the risks brought by online exploratory interactions and delivers a paradigm with great potential for real-world USV implementations.