An LLM-Driven Hybrid Online-Offline MARL Architecture for AUV Cluster Network to Enable Smart Underwater Target Encirclement

Abstract

With the rapid advancement of underwater networking and multi-agent reinforcement learning (MARL) technologies, autonomous underwater vehicle (AUV) cluster networks have emerged as a promising framework for enabling smart underwater missions, particularly in cooperative target encirclement. However, training MARL policies in such environments faces a critical challenge: acquiring large-scale interaction data is infeasible due to the severely constrained communication bandwidth, especially for online MARL frameworks. Therefore, achieving fast policy convergence with limited data collection is essential for practical deployment. This article proposes an LLM-Empowered Hybrid Training (LLM-EHT) architecture for MARL, which leverages the reasoning and generative capabilities of large language models (LLMs) to facilitate the transition from online to offline MARL under limited-sample conditions. Specifically, the proposed architecture first collects a small amount of online interaction data. It then employs an LLM to synthesize an offline dataset, guided by a dedicated alignment loss that enforces consistency between policy actions and LLM-generated references. This process significantly accelerates MARL policy convergence. Building upon LLM-EHT, we further develop a smart and cooperative underwater target encirclement scheme, incorporating a scalable state space representation method and a smart target encirclement policy to ensure robust, efficient, and scalable operation. Evaluation results showcase that the proposed scheme significantly reduces the required sample size while achieving faster convergence and maintaining stable encirclement effectiveness.