Multi-agent Reinforcement Learning for Cooperative Control of Multiple Unmanned Surface Vehicle in Neutralizing Threat Vessels

Abstract

This study proposes a cooperative capture system in which two allied unmanned surface vehicles (USVs) jointly deploy a capture net to intercept noncooperative vessels in naval environments. To overcome the limitations of traditional rule-based methods subjected to environmental uncertainties, a multi-agent reinforcement learning (MARL) framework is developed within a high-fidelity, unity-based simulation environment incorporating the Gerstner wave and wind disturbance models. Each agent utilizes an egocentric polar coordinate observation space; distance values are normalized via a nonlinear rational function to prioritize close-range sensitivity, while angular discontinuities are resolved through trigonometric transformations. The Multi-Agent POsthumous Credit Assignment algorithm is adopted within a Centralized Training with Decentralized Execution architecture, enabling robust strategy learning through posthumous credit assignment even when agents exit the environment. The reward structure is designed hierarchically, combining sparse event rewards for capture success with dense continuous rewards based on formation maintenance, target approach, and heading alignment. Quantitative experimental results demonstrate that the proposed MARL-based approach significantly outperforms the modified Line-of-Sight guidance law, achieving a 65% capture success rate and a three-fold reduction in formation collapse under dynamic sea conditions. This study confirms that the proposed framework effectively enables autonomous USVs to learn complex, coordinated maneuvers for physical maritime interdiction, providing a robust solution for securing maritime security.