CA-MARL: credit-aware multi-agent reinforcement learning for USV pursuit–evasion missions under complex maritime disturbances
TL;DR
A Credit-Aware Multi-Agent Reinforcement Learning (CA-MARL) framework for multi-USV pursuit–evasion missions, which features two key innovations: a Residual Self-Attention module that adapts to varying fleet sizes through permutation-invariant attention, and a Mixed Credit Assignment module that enhances centralised value estimation with decentralised branches.
Abstract
Multi-unmanned surface vehicle (USV) pursuit–evasion missions in maritime environments presents significant challenges due to dynamic ship populations, high-dimensional observations, and the gap between idealised simulations and real-world maritime physics. To address these challenges, we propose a Credit-Aware Multi-Agent Reinforcement Learning (CA-MARL) framework for multi-USV pursuit–evasion. The framework features two key innovations: a Residual Self-Attention module that adapts to varying fleet sizes through permutation-invariant attention, and a Mixed Credit Assignment module that enhances centralised value estimation with decentralised branches. Moreover, to bridge the simulation-to-reality gap, we develop a high-fidelity 3D virtual platform using Unity3D that incorporates maritime factors, such as hydrodynamics and wave disturbances, which are typically overlooked in USV simulations but critical for maritime operations. Experiments demonstrate that our method achieves superior coordination, sample efficiency, and policy robustness compared to existing baselines, providing a credible foundation for deploying MARL policies in realistic multi-USV scenarios.